Category Archives: Programming

2018 Early voting analysis

You know you’re in a programming town when this gets run on your local TV station;s website:

Full methodology:

Latitude and longitude coordinates for 2018 early voting locations were obtained from the State Board of Elections and Ethics Enforcement’s lookup tool by using a Python script.

Coordinates were not available for 2014 through this tool, so the bulk of these locations were generated using the U.S. Census geocoder tool. Addresses that could not be matched were manually researched and recorded using Google’s geocoder tool.

In 30 North Carolina counties, there were no changes in early voting locations between 2014 and 2018, so these counties were omitted from the analysis. This left 580 sites for the two midterm elections. Voters in these counties were also omitted from this analysis, leaving 6,433,969 active and inactive voters, both of which are eligible to cast ballots, according to state elections officials.

While some early voting locations may have been relocated due to the impact of hurricanes Florence or Michael, this analysis considered only the original early voting locations approved by local elections board and the state board.

Latitude and longitude coordinates were then matched to active and inactive registered voters on addresses, city, county and ZIP using MySQL database software. The query failed to match the addresses of 145,645 voters, a 97.7 percent match rate.

We used the free application programming interface (API) from the Open Route Service to generate isochrones – polygons for geographic information systems used to determine driving distances radiating outward from a point source. Isochrones were generated programatically using a Python script.

Open Route Service limits queries through its API to 10 shapefiles at a time. The service also limits total API queries to 2,500 a day.

Due to these limitations, the Python script runs queries for each site four times to produce a geojson feature collection with shapefiles at half-mile intervals from 0.5 to 20, with each polygon describing a driving distance range.

For example, a point that appears in the isochrone with a mile value of 5, but not in an isochrone with a mile value of 4.5, is within 4.5 and 5 miles from the early voting location.

Voter registration data, in CSV format, are loaded into the database, and a separate Python script was used to import the isochrone geojsons using ogr2ogr and its pygdaltools wrapper with a Python script.

SQL queries can then generate mile values for each isochrone intersecting each voter, by county. By deduplicating the table based on the voter and keeping the smallest value, we can find the closest site and distance for each voter in 2014 and 2018.

We then used database software to calculate the change in distance from the closest voting location in 2014 and the closest early voting location in 2018 for every active and inactive voter.

Because the driving distances were limited to 20 miles from each voting location, 62,325 voters could not be matched with either a 2014 or 2018 isochrone because they were outside the 20-mile range. This amounts to less than 1 percent of the registered voters in the study for which the difference in driving distance could not be calculated.

Floating Video Killer

If you’re annoyed by the floating, screaming video player that haunts CBSNews.com pages, here’s a Greasemoney script I wrote to kill it:

// ==UserScript==
// @name         CBSNews Floating Video Killer
// @include      https://www.cbsnews.com/*
// @grant        none
// @noframes
// ==/UserScript==

setInterval(function handler() {
  
  // front page
  var frameDiv = document.querySelector(".embed__content--draggable");
  
  // article
  if (frameDiv == null) frameDiv = document.querySelector(".media-block");
  
  if (frameDiv != null)
  {
    // are we looking at the floating frame, or the header player?
    var floated = false;
    var classList = frameDiv.className.split(/\s+/);
    for (var i = 0; i < classList.length; i++) {
      if (classList[i] === 'floating') {
          floated = true;
        break;
      }
    }
    
    // don't touch the header player's button
    // (or you'll never be able to play it at all!)
    if (floated) {
      var element = document.querySelector(".player-overlay__button");
      if (element) {
          element.click();
      }
    }
  }
  
}, 1000);

(really just posting this here so I can share it between my own computers)

eFail!

This is kindof hilarious…

That brings us back to last week, and the release of Efail. The hack is simple and brilliant: It uses the fact that your email client thinks it’s a web browser. An attacker sending mail can steal the content of secret messages you may have sent or received. It works like this: An email client running OpenPGP (the current standard of PGP) or S/MIME decrypts messages when it receives them, and since the clients are also web browsers, they fetch things from the web for displaying them to you in the email you open at the same time. So what if you happened to open an email, which decrypts whatever message it may have inside, even a hidden one, while the same email also tells your email client to fetch an image off the web whose name is now the entire contents of a message it just decrypted? It would just do it, invisibly, sending the now easily readable message anywhere on the net without you ever knowing it happened. Sure, an image named “Meet me at the park on Sunday at 3 a.m. and we’ll make plans from there come alone.jpg” would never load on your screen, but you’ll have invisibly asked for it, and that ask will now be recorded in whatever computer out there the person who sent the mail wanted it recorded on. And that mail could have just as easily said it was from your spouse or boss as God or Santa Claus.

EMail is fundamentally broken.

Alas. I like email.

Source: Email Hackers Are Winning – The Atlantic

Throw Them All Away

Schneier on Spectre and Meltdown:

The security of pretty much every computer on the planet has just gotten a lot worse, and the only real solution — which of course is not a solution — is to throw them all away and buy new ones.

“Throw it away and buy a new one” is ridiculous security advice, but it’s what US-CERT recommends. It is also unworkable. The problem is that there isn’t anything to buy that isn’t vulnerable. Pretty much every major processor made in the past 20 years is vulnerable to some flavor of these vulnerabilities. Patching against Meltdown can degrade performance by almost a third. And there’s no patch for Spectre; the microprocessors have to be redesigned to prevent the attack, and that will take years. (Here’s a running list of who’s patched what.)

This is bad, but expect it more and more. Several trends are converging in a way that makes our current system of patching security vulnerabilities harder to implement.

Spectre and Meltdown are pretty catastrophic vulnerabilities, but they only affect the confidentiality of data. Now that they — and the research into the Intel ME vulnerability — have shown researchers where to look, more is coming — and what they’ll find will be worse than either Spectre or Meltdown. There will be vulnerabilities that will allow attackers to manipulate or delete data across processes, potentially fatal in the computers controlling our cars or implanted medical devices. These will be similarly impossible to fix, and the only strategy will be to throw our devices away and buy new ones.

Schneier is rarely optimistic when it comes to security issues. But I’d never bet against him being right, either.

Generating fantasy maps

Here is an interesting article about the algorithms used to generate ‘fantasy’ maps like this one:

This is far beyond the programs I wrote to generate terrain on my Amiga 500, way back in ’87. I was content with square grids, no ‘erosion’, no labels, etc.. But, I did draw them in color, with nice isometric 3D projection!

VBScript!

I’ve been fighting with a little script I wrote to massage some text for me. It’s a very simple task so I thought VBS would be adequate (read each line of text, if there’s a special bit of text in the line capitalize everything to its right – no big deal). I know I can use C# and JavaScript if I use Windows’ Powershell, but I didn’t think VBS would have any trouble with it. And it didn’t. But I learned something ridiculous about VBS in the process.

Here is a very simple VBScript program.

Function timesTwo(ByRef inParam)
    inParam = inParam * 2
    timesTwo = 1
End Function

i = 10

WScript.Echo "i = " & i
timesTwo(i)
WScript.Echo "i = " & i
z = timesTwo(i)
WScript.Echo "i = " & i

Here I have a simple function called ‘timesTwo’ that multiplies its input parameter by two and returns a value of 1. Since I declared the input of the function “ByRef”, the input is passed “by reference”, which basically means the function can change the value of the parameter. If I pass it a variable with a value of 5, the variable should have a value of ten when the function ends. Magic.

If the parameter was declared “ByVal” (“by value”), the function would receive a copy of the input parameter, which it could modify as much as it wanted, but changing the copy wouldn’t change the value of the original parameter. But I didn’t declare it that way…

The code below that function tests the function. It…

  1. initializes a variable called ‘i’ to a value of 10 (i = 10)
  2. prints the value of i (WScript.Echo "i = " & i )
  3. calls the function, ignoring its return value. (timesTwo(i))
  4. prints the value of i
  5. calls the function again, putting its return value in a variable ‘z’. (z = timesTwo(i))
  6. prints the value of i

So, what do you think we should get?

i = 10
i = 20
i = 40

That seems reasonable. Start with i = 10, print i, call the function to double i, print i, call the function to double i again, print i.

But if that worked, this would be a truly boring blog post!

No, what we actually get is:

i = 10
i = 10
i = 20

Why? Well, I don’t know exactly why. But, the effect is: if you ignore the return value of a function, the parameter you pass is actually passed by value (ByVal), not by reference.

Pass by value:

timesTwo(i)

Pass by reference:

z = timesTwo(i)

What. The. Fuck.

Why would that difference in behavior ever be useful, let alone expected ?

I should’ve used JavaScript.

Dynamic Camouflage

There’s a specific technique for finding the optimal solution to complex problems (typically mathematical or computer programming problems) by breaking them into simpler sub-problems – each of which only needs to be solved once, optimally – and then combining the results of those sub-problems to arrive at the final solution. It’s called “Dynamic Programming“. Sounds sexy! But when you get into it, you notice that there’s really nothing dynamic about it. When you start down the DP path for a problem, you don’t find yourself in an exciting world of fast and forceful – dynamic! – things to think about; rather, it’s more about carefully and deliberately rethinking the problem at hand until you find a way to flip it on its head, or turn it inside out, or maybe even solve it backwards.

And the name doesn’t actually refer to computer programming. It’s about mathematical programming: finding the best possible solution from the set of all possible solutions, given some definition of ‘best’. That DP is often used in computer programming today is coincidence.

So where does the name come from?

The man who came up with the name, Richard Bellman, explains:

“I spent the Fall quarter (of 1950) at RAND. My first task was to find a name for multistage decision processes. An interesting question is, Where did the name, dynamic programming, come from? The 1950s were not good years for mathematical research. We had a very interesting gentleman in Washington named Wilson. He was Secretary of Defense, and he actually had a pathological fear and hatred of the word research. I’m not using the term lightly; I’m using it precisely. His face would suffuse, he would turn red, and he would get violent if people used the term research in his presence. You can imagine how he felt, then, about the term mathematical. The RAND Corporation was employed by the Air Force, and the Air Force had Wilson as its boss, essentially. Hence, I felt I had to do something to shield Wilson and the Air Force from the fact that I was really doing mathematics inside the RAND Corporation. What title, what name, could I choose? In the first place I was interested in planning, in decision making, in thinking. But planning, is not a good word for various reasons. I decided therefore to use the word “programming”. I wanted to get across the idea that this was dynamic, this was multistage, this was time-varying. I thought, let’s kill two birds with one stone. Let’s take a word that has an absolutely precise meaning, namely dynamic, in the classical physical sense. It also has a very interesting property as an adjective, and that it’s impossible to use the word dynamic in a pejorative sense. Try thinking of some combination that will possibly give it a pejorative meaning. It’s impossible. Thus, I thought dynamic programming was a good name. It was something not even a Congressman could object to. So I used it as an umbrella for my activities.”

Rain girl

This is one of my favorite pics – a girl walking in the rain outside our first house. I’ve posted it previously, a couple of times.

I came up with an implementation of Aaron Hertzmann’s “painterly” non-photorealistic rendering algorithm. This recreates the source image by using simulated brush strokes of varying size, color and shape. It’s not a blurring or filtering of the source; it’s a recreation made in much the same fashion that a human painter would copy a picture: look at the source, choose a color, make a stroke which follows that color. Repeat until you’re done. As with a human painter, it starts with large brushes to get the background and large areas filled, then it refines the image with progressively smaller brushes.

This video is the algorithm in action, drawing the strokes (20 per frame, to save time). I told it to use short strokes.

The music is called “One Bird”. That’s me.

Here’s one of a pic of Adrian Belew:

Longer, thinner brush strokes.

And, Tricksey:

Tricksey, Painterly

Drat Of The Day

Which is the minimum of the two:

A: -3.4028235e+038
B: 1.1754944e-038

That depends on how you define minimum. If you want the smallest magnitude, it’s B. B is so close to 0 it might as well be 0 for most practical uses (0.000000000000000000000000000000000000011754944). But A is smaller if you think ‘minimum’ means ‘leftmost on the number line’. A is so far to the left of B there aren’t enough pixels on all the monitors in the world to draw that number line.

In C++ the result of std::numeric_limits<float>::min() is B: the smallest value a ‘float’ can store is B.

But the most-negative number a float can store is A. And A is std::numeric_limits<float>::max().

So, if you are trying to limit the results of a function to the range of values that a float can hold, and you do this:

double result = doSomething();
double t = max(std::numeric_limits::min(), result);
t = min(std::numeric_limits::max(), t);
return (float)t;

You’re going to very surprised when you discover that you never see a value smaller than 1.1754944e-038. You’ll never even get 0.

What you really needed to write is this:

double result = doSomething();
double t = max(-std::numeric_limits::max(), result);
t = min(std::numeric_limits::max(), t);
return (float)t;

But, that trickery only applies to floating point types. The smallest magnitude you can store in an int is 0. But std::numeric_limits<int>::min() is -2147483648. That’s the most-negative value, not the value with least magnitude.

And now you know.