What is a Data Scientist?

Today on Quora, someone asked, "What are some software and skills that every Data Scientist should know?". I wrote the following as a response, reflecting on my current position and the role I play.

---

I started adding post-it notes with sub-titles to my name/title tag on my cube, as sort of a joke regarding the question, "What is a Data Scientist?".

Here's the current list:

• [Client] Analyst
• [Product A] Analyst
• Financial Analyst
• Sales Analyst
• Contract Analyst
• Quality Assurance Analyst
• Call Center Analyst
• Data Surgeon (aka, data mining with the intent to figure out what's wrong)
• Data Diagnostician (alternative of above, maybe with no details to examine)
• [Product B] Analyst
• Database Developer
• Bug Finder (as in software bugs)

So it would appear from this list that there isn't a lot of data science going on. And that's partially true.

Each of our clients has its own relational database, so we do "meta-queries" to access them one by one in order to answer a question. That's sort of data science like. Eventually, though, we're going to have one master database with all clients that will cascade into individual databases. So our "meta-queries" will be obsolete.

We deal with a lot of "big data" too, but it's usually not that big of a deal. Even with relational databases, it's okay. Some queries may take a little longer (30-60 minutes), but that's rare. We have some machine learning tasks that pull in massive training data sets, so at that point you have to be more careful about "big data" problems like running out of RAM or disk space. But it can be handled, and rather simply.

What I really wish I could do more of is machine learning, and while I've accumulated several ideas that would enhance products or help us make better decisions in the year I've been a Data Scientist, these other tasks take up most of my day.

In the end, I write a lot of SQL, use the Linux command line moderately, and report on data in Excel spreadsheets. I use Python occasionally to write scripts. And I'm always learning something new (new SQL techniques, Python libraries, Linux command line tools, etc.).

An Amended Quote

Today on The Writer's Almanac, Garrison Keillor read a quote by Gustave Flaubert:

"I spent the morning putting in a comma and the afternoon removing it."

I thought this could use some amending for programmers:

I spent the morning putting in a parenthesis and the afternoon removing it.

Or for you SAS programmers:

I spent the morning putting in a semicolon and the afternoon removing it.

How about SQL:

I spent the morning putting in a column and the afternoon removing it.

Or if you write HTML:

I spent the morning putting in an escaped ampersand and the afternoon removing it.

Why not Python:

I spent the morning importing a library and the afternoon removing it.

Essentially, though, these all boil down to:

I spent the morning putting in (an) arbitrary character(s) and the afternoon removing it.

For all you programmers during this holiday season: May your parentheses be matched (your quotes too!), special characters escaped, and code executable on the first run.

An Attempt to Answer a Quiz Question

Kyra heard the NPR Weekend Edition Sunday puzzle on the radio the other day: "Think of a common French word that everyone knows. Add a 'v' (as in 'violin') to the beginning and an 'e' at the end. The result will be the English-language equivalent of the French word. What is it?"

I thought, well, I don't know French, but I do know how to write a Python program that can look up the answer. Unfortunately, the puzzle appears to be a bit of a trick. I thought as much as I was writing the code. I was thinking, "I bet some people are going to be mad about the answer, because the question was worded in such a way as to be tricky."

At any rate, the code definitely does not result in the right answer, but it was a good exercise for me. First, I had to find two dictionaries robust enough to have a lot of words in English and French. Then I had to find dictionaries: I located the Debian operating system dictionary on my local computer, did a search in the package manager Synaptic for the French equivalent and any other English dictionaries that might help, and downloaded both. (The English alternative is labeled "insane" for its size, which "possibly contains invalid words (as well as words that are very uncommon)." [1])

Next I had to convert certain French characters into their English equivalent. To answer the question, I looked up all English words that start with "V" and end with "E", then removed those two letters, looking up the "word" in the modified French dictionary. It didn't work, as can be told from its output:

a
a
ah
aire
alu
an
ange
ares
as
er
erg
es
es
il
o
t

Yep, those are definitely not the answers. Some, when you add V and E back, aren't  even really English words. "Vte"? What is that? Oh, and I think A is listed twice due to the missing diacritics.

For anyone who's interested, I posted the code online. (It's not pretty posting a lot of code on Blogger, so here it is in a document.) Keep in mind I'm a new programmer, so I might not have done this task very elegantly. It is a short piece of code, though, and it could have worked.

One other note: If you attempt to run it, it may not work on your system. It's designed for Debian GNU/Linux, as it relies on dictionaries in a Debian system and it uses the "/" as the directory delimiter. Additionally, the "insane" dictionary makes it run a bit slow, and I did nothing in the code to help with that, like buffering the file, so you could run out of RAM or peak the CPU. It works on my system, which is all that mattered to me at the time.

Of course, a more robust program should be more careful and actually take user input in some fashion. Perhaps look up words that with other characters or use different languages. As it is, it's more of a "script" that does one thing and non-interactively. In any case, it doesn't even return the "right" answer!

[1] http://packages.debian.org/squeeze/wamerican-insane

Update in May 2012

Last year around this time I had decided to take a risk and leave my position at the University of Wisconsin Medical Foundation for a position at the UW School of Medicine, where I was hoping to be more involved with research. This endeavor hasn't turned out as I expected, so I've decided to move on. Starting later this month, I'll be working with CPM HealthGrades, a marketing firm that is offering more and more data services to their customers.

I accepted the position at CPM HealthGrades as a Data Scientist. This is a pretty cool opportunity in a relatively new field of data analysis, using some of the same technologies behind IBM's Watson computer, the computer that bested two top-rank Jeopardy winners. I'll be learning the programming languages Python and R, and I'm really excited to put these skills to commercial and research use.

As you may know, I attended the SAS Global Forum conference last month, and I plan to keep up with the SAS community as well, by writing papers, presenting, staying in touch with user groups, and maybe even writing a book. So I plan on keeping my SAS site up-to-date; however, I may not be developing much new. Perhaps I'll come out with a Python or R site to go along with it, with translations of code back and forth between the languages. Heck, maybe I'll write a SAS macro to write Python, or a Python module to write SAS. We'll see! Stay tuned for more details!

SAS Macros versus Python Functions, Part 1

I recently started reading Dive Into Python, a primer to the programming language Python (which I converted to MOBI and loaded onto my Kindle - available here), and I was delightfully surprised how similar functions are to the SAS macro. Here are some examples:

First, the very first line of a function or macro, that defines it as a function/macro, is very similar. In Python, one would write:

def buildConnectionString(params):

In SAS, it is very similar:

%macro buildConnectionString(params);

The terms def and %macro begin the code by stating what follow: a function definition or a macro. The "buildConnectionString" is the name of the function/macro, and enclosed in parentheses are the function/macro parameters or arguments, separated by commas.

Second, it is often a good idea to have some documentation on the function/macro. In Python, the documentation is stored in a doc string; in SAS, a very short and not as useful description string is available:

def buildConnectionString(params):
    """Build a connection string from a dictionary of parameters.

    Returns a string."""

SAS:

%macro buildConnectionString(params) / des="Build connection string. Returns string.";

UPDATE: I'm not sure where I learned the 40-character limit mentioned below, but I just read in the SAS help documentation that the limit is 256 characters. Nonetheless, reading the description is still limited by screen space, so the alternatives I mentioned here are still very useful.

Unfortunately, SAS macros are limited to 40 256 characters in the description. This can be worked around by offering HELP as a valid value to the first argument, which then outputs additional information about the macro. My macros are documented in a spreadsheet, and I have a macro (ArgList) that reads in the spreadsheet and outputs additional details about the arguments when I need it. One day, I would like to convert all of my macros to use a standard header that contains the information about the arguments, its purpose, and other details in a standard way so I can modify the ArgList macro to read the macro itself instead of external documentation. In Python, though, the doc string can be viewed when referencing the function, on the fly. That's incredibly useful, and much more flexible than SAS.

One area of difference is the syntax and the meaning of space: In all Python code, space has meaning, whereas in SAS code in general, it does not. Even so, I typically indent my SAS code just like Python code requires (in most cases - I try to maintain readability, too).

Once I learn more about Python, I'll post more comparisons!