Reimplementation of scraping

As the scraping scripts we started out with had a rather high run time and were organised pretty complicated, the whole thing was rewritten. The spiders use the central team (https://igem.org/Team_List?year=%d) and results (https://igem.org/Results?year=%d) page per year to go through all the teams at once instead of opening, editing and closing the JSON results file for every team. Within each team page HtmlXPathSelectors are used to generate the single values. As this of course depending on the exact same page structure, nothing but the two central pages can be easily scraped using these detailed selectors.

Wiki Scraping

We wrote a pipeline to collect and print the data for every team properly. Thus proper script exiting and output file specification via the command line were easier to implement. The track, which was missing until now was included in the scraping process.
Furthermore several bugs were fixed. We had encountered problems regarding whitespaces, espacially the dublication of teams when their name contains a blank space. We also took care of the proper construction of all the spider objects.

Data conversion to R

The JSON file needs to be converted to R compatible data for the analysis. The target file contains one dataframe for all single value parameters and one list for all multiple value parameters / gib text contents. Here unique naming of the teams was achieved by combination of name and year. A full liste of which parameters and contents were converted is displayed in table 4.1.

Table 4.1: Parameters generated so far.

Data frame parameters	List elements
Numerical: year students count advisors count instructors count regional awards count championship awards count biobrick count Character strings: region wiki url (Team overview page)	year character vector of regional awards character vector of championship awards parts range advisor names project title abstract

First shiny app implemented

A first shiny app that can display the numerical data in a scatterplot was created. Besides shiny it also uses the RCharts package for more aesthetic graphs.

Different concepts for shiny files

Since there are two different concepts of how to implement the page surrounding a shiny app, we tried out both.
See week 20 for the final solution we chose for building our apps.

R builder functions for html

Files needed:

• ./server.R
• ./ui.R

This solution turned out to be very complicated due to the combination of directly accessible html tags, those which need the tags() function and styling rules within the ui.R. Thus using the other concept seems more intuitive.

Separated html code

File needed:

• ./server.R
• ./www/index.html

Here the app is built directly in html and css syntax as any other basic homepage. This on the other hand became complicated, when integrating the server output in the app.

Second app implemented

The second shiny app we impelmented was the timeline app. It counts teams, awards or students within each year and puts the numbers in a bar plot. For the plot we again use RCharts but this time with nvd3. In order to prevent errors we had to add empty values to the data since nvd3 can't handle incomplete datasets.

First hompage draft

A simple draft of the homepage was implemented using basic html and css tags.
This page is supposed to either become integrated in our wiki or to be a stand alone homepage.

Layout

page wrapper

App integration

In order to integrate the apps, the central iframe was used as target for the links in the menu bar. The link would then for example look like this:

<a href="apps/factsfigures/" target="center-frame">Facts & figures</a>

Creation of several design suggestions

During a meeting we decided to create three design proposals, so that the team could vote which one they like best. Out of the three one should be a modern design similar to other web pages, having a virtual third dimension. With regard to the Hitchhiker's Guide to the Galaxy we wanted one "spacy" design and lust but not least one alchemical / old fashioned design to fit our team logo.
Note: Since we finally used the same design as on our wiki, it's not worth showing the other design proposals or the introductory texts at this point. These can all be found on our wiki.

Draft of filtering options

The drafts for most of the filtering functions were implemented. Table 1 gives an overview of the different parameters and their basic filter design.

Table 1: Overview of all filters.

Parameter	Type	Options*	Status
Year	numeric	2007-2012	draft
Region	string	levels	draft
Track	string	levels	draft
Students	numeric	0, 5, 10, 15, 20, >20	draft
Advisors	numeric	0, 2, 5, 10, 15, >15	draft
Instructors	numeric	0, 2, 5, 10, 15, >15	draft
Biobricks	numeric	0, 5, 10, 20, 50, 100, 200, >200	draft
Championship	character vector	levels	draft
Regional	character vector	levels	draft
Medals	string	bronze, silver, medal	missing
Score	numeric	0-100 (steps of 10)	draft
Abstract	binary	provided/missing	missing
* Levels means all possible values the parameter can have.

The general code for filtering a numerical range parameter X using R looks like this:

FilterForX <- function(data) {
	if (input$FILX_min == ">max") data <- data[-which(data$X < max),]
	else if (input$FILX_max == ">max") data <- data[-which(data$X < input$FILX_min),]
	else data <- data[-which(data$X < input$FILX_min | data$X > input$X_max),]
	return(data)
}

The general code for filtering a single string parameter Y using R looks like this:

FilterForY <- function(data) {
	matchY <- rep(0, times=length(data$Y))
	for (i in 1:length(input$FILY)) {
		matchY[which(data$Y == input$FILY[i])] <- 1
	}
	data <- data[-which(matchY == 0),]
	rm(matchY)
	return(data)
}

For each filter the corresponding selection elements (max, min or multiple selection) are added underneath each other below the plot of the app. They are grouped as teams success and team composition.

Implementation of Topics app started

For the topics and the methods app we had several ideas on how to structure them. Besides a timeline of the hottest topics or histogramm view of the topics distribution the most important feature we had in mind was the direct search for the meshterms we extracted.
Pseudocode for the corresponding filtering functions and a draft of the interface were written, but the workload in the other projects was too big to actually create a functional app.

Filters fixed

The drafts for most of the filtering functions had already been implemented, but ended up non-functional. This was due to data removal using empty vectors, which produces a severe error in R. Table 18.1 gives an overview of the different parameters and their basic filter design.

Table 18.1: Overview of all filters.

Parameter	Type	Options*	Status
Year	numeric	2007-2012	final
Region	string	levels	final
Track	string	levels	final
Students	numeric	0, 5, 10, 15, 20, >20	final
Advisors	numeric	0, 2, 5, 10, 15, >15	final
Instructors	numeric	0, 2, 5, 10, 15, >15	final
Biobricks	numeric	0, 5, 10, 20, 50, 100, 200, >200	final
Championship	character vector	levels	working
Regional	character vector	levels	working
Medals	string	bronze, silver, medal	missing
Score	numeric	0-100 (steps of 10)	final
Abstract	binary	provided/missing	missing
* Levels means all possible values the parameter can have.

The general code for filtering a numerical range parameter X using R after this update is displayed below (changes are displayed in red).

FilterForX <- function(data) {
	if (input$FILX_min == ">max") data <- data[-which(data$X < max),]
	else if (input$FILX_max == ">max" & input$FilX_min != "min") data <- data[-which(data$X < input$FILX_min),]
	else if (input$FILX_max == ">max" & input$FilX_min == "min") return(data)
	else data <- data[-which(data$X < input$FILX_min | data$X > input$X_max),]
	return(data)
}

The general code for filtering a single string parameter Y using R now looks like this:

FilterForY <- function(data) {
	matchY <- rep(0, times=length(data$Y))
	for (i in 1:length(input$FILY)) {
		matchY[which(data$Y == input$FILY[i])] <- 1
	}
	delete <- which(matchY == 0)
	if (length(delete) != 0) data <- data[-delete,]
	rm(delete)
	rm(matchY)
	return(data)
}

The code for matching the awards is slighly more complicated, since the awards are saved in the data-list and we want to reduce the data-frame. This is the pseudocode for the functions:

iterate through all awards to be included {
	(re-)set a vector for deleting those items from the data-list
	iterate through all the teams in the data-list {
		if the award matches one of the team awards:
		- expand the vector for the teams to keep with the name
		- expand the vector for the teams already matched
	}
	delete those teams from the data-list, that were already positive in order to save runtime
}
reduce data-frame according to the matching result

All the filter functions and corresponding layout elements were put together in one file to easily copy and paste them to the different apps.

Restyling filters including javascript and css

In order to have the whole apps in a compact layout, the filters were put into a notebook. To navigate and style this javascript and css had to be included. This works through the simple "includeHTML()"-function shiny contains.
The tab heading are implemented as a unordered list containing links. These links activate a javascript, which shifts between the containers displayed in the notebook. This works by first hiding all containers in that area and then showing the one that was specified as parameter, when calling the function from the link.
The style of the tabs and the notebook were adjusted to match the border color and radius of the bootstrap styled elements.

Adjustments to page design

To allow for proper presentation in meetings, the design of the test-homepage was adjusted to current wiki design.

Expansion of Filters

The last filters missing (medal and abstract), another filter for team names and one for the information content of the abstract were implemented. Additionally the selection ranges of some filters were adjusted. See table 23.1 for the final filter setups.

Table 23.1: Overview of all final filters.

Parameter	Type	Options*	Status
Year	numeric	2007-2012	final
Region	string	levels	final
Track	string	levels	final
Students	numeric	0, 5, 10, 15, 20, >20	final
Advisors	numeric	0, 2, 4, 6, 8, 10, 12, 14, >14	final
Instructors	numeric	0, 2, 5, 10, 15, >15	final
Biobricks	numeric	0, 5, 10, 20, 50, 100, 200, >200	final
Championship	character vector	levels	final
Regional	character vector	levels	final
Medals	string	levels	final
Score	numeric	0-100 (steps of 10)	final
Abstract	binary	all/only provided	final
Information content	numeric	0, 0.4, 0.45, 0.5, 0.55, 0.6	final
Team name	string	variable entry	final
* Levels means all possible values the parameter can have, which is either generated automatically or manually.

Abstract

The filter for the abstract only checks, whether an abstract was submitted or not. See the pseudocode:

if user wants to see all teams return full data set
else {
	iterate through all row names in the data-frame {
		if the abstract of the team "row name" in the data-list matches the string "-- No abstract provided yet --" save the name to a vector
	}
	delete all teams without abstract
	return the reduced data set
}

Since the information content is also related to the abstract, they are put together in one tab of the filter notebook. As it turned out that the information content always is in a very small range around 50%, the possible selections were chosen accordingly. The filter function is similar to those of any other numeric parameter, where the only difference is that all choices are numeric and thus the different cases of minimum and maximum selection can be skipped.

Medal

The medal filter is just the same as the filters for track or region, since it is also saved in the data-frame.

Team Name

For filtering the team name a text entry box was introduced, together with a javascript, that checks on the entered search term. A list of all team names was created and converted to a javascript array using R and its package RJSONIO. The javascript code then compares the entered term(s) with this array and changes the text color to either red (no exact match) or black (exact match). To allow for multiple name entries comma-serapated terms can be entered and are automatically separated by the javascript for color changing and by the filter function.
The pseudocode of the filter funciton looks like this:

split the input string at every comma
iterate through all strings {
	find all teams having the string in their name and keep their ids
}
reduce and return dataset

Team table output

All the apps so far only displayed data, but with no option to see which teams contrbuted to each portion of the graphs. Thus a table output was implemented, that shows the teams' names, years and wiki links. To allow for better overview, one can select how many tams should be display (none, 5, 10, 20, 50, 100 or all) and in which order. This can be done either by the score, displaying the best teams at the top of the list, by year starting with most recent teams, or alphabetically. The data-frame displayed always represents the fully filtered data. Thus the user can narrow the number of teams down using the filters and can then go directly to the corresponding wiki pages.

Update of methods

The list of the methods was slightly updated and the clusters were removed, since they turned out to be not very intuitive. In order to find methods, that are actually mentioned in abstracts we went through those of teams who were successful in either the new application track, experimental measurement aproches and other practice associated directions. The updated list is displayed in table 22.1.
After rerunning the postprocessing of our data sets the top 10 methods list actually contained 10 methods of which many were newly added.

Table 22.1: Updated methods list without clusters

Old	New	Removed
Fusion Proteins, Primer Design, preparation of DNA, Restriction Digestion, Insert preparation, cell fractionation, cell counting, PCR, interaction chromatography, purification, Gel extraction, Ligation, Transformation, FRET, DNA extraction, patch clamp, Northern Blot, Southern Blot, Bioinformatics, ELISA, Chromatography, flow cytometry, X-Ray-crystallography, NMR, Electron microscopy, Molecular dynamics, coimmunoprecipitation, Electrophoretic mobility shift assay, size determination, gel electrophoresis, macromolecule blotting and probing, immuno assays, phenotypic analysis, imaging, spectroscopy, spectrometry	traditional cloning, Gibson, bacterial artificial chromosome, protein interaction, FISH, assembly line, Sequencing, Microarray, computer simulation, drug production, encapsulation, genome deletion, mutagenesis, next generation sequencing, image processing, kill switch, Western Blot, Mass spectrometry, southwestern blot, fluorescence microscopy	cloning, DNA sequencing, DNA Microarray, arrays, Western blotting, southwestern blotting

Bug fixing

We noticed, that not all of the award filters gave the right number of teams. This was due to false unlisting in the JSON to RData conversion script. After fixing this bug and after removing the false runner up awards in 2011 all the awards were finally correct.
By converting the methods to lower case before matching them methods missing beforehand, as for example Gibson and FRET, could also be detected.

Design adjustments

In order to have a nice layout when integrating the apps into a page resembling our wiki, a text box gicing a short introduction to the corresponding app was put next to the major input elements. This top section is now optically separated from the results area and all apps fit nicely into the minimum width of the iframe, which is 900px.

Chart details

As most of our charts are only displaying discrete values we wanted to get rid of the extra decimal positions in the axis labeling. This was achieved by adding a short piece of javascript in the corresponding nvd3 plot parameters. Since the number of tick marks was not reduced by the rounding of the numbers this produced repetitive tick marks if only small values were displayed. To avoid this the height and the position of the plot were changed depending on the number of ticks to be displayed. As the final detail browser compatibility tags were added to the css and axis labels were added to the plots.