{"id":27063,"date":"2020-06-07T20:56:58","date_gmt":"2020-06-08T00:56:58","guid":{"rendered":"https:\/\/www.bu.edu\/econ\/?page_id=27063"},"modified":"2025-09-29T10:28:50","modified_gmt":"2025-09-29T14:28:50","slug":"research-computing","status":"publish","type":"page","link":"https:\/\/www.bu.edu\/econ\/research\/research-computing\/","title":{"rendered":"Research Computing &#038; Data Resources"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">Prof. Marc Rysman has given a seminar about why and how to use the cluster every few years. Here are the slides, with some minor updates:<\/span><\/p>\n<p><a href=\"\/econ\/files\/2020\/09\/SCC-Talk-2020-09-11.pdf\">Prof. Schmieder&#8217;s Slides on Research Computing<\/a><br \/>\n<a href=\"\/econ\/files\/2020\/07\/introToSCF2020-handout.pdf\">High Performance Computing for BU Economists (Previous version of slides) <\/a><\/p>\n<p><b>Important<\/b><span style=\"font-weight: 400;\">: The slides describe how to obtain access to the cluster.<\/span><\/p>\n<p>The faculty RCS liaison is Jean-Jacques Forneron. Graduate students that want access to the cluster should email <a href=\"mailto:miyauchi@bu.edu\">JJ<\/a>.<\/p>\n<p>There is also an RCS Student Ambassador. For help, you can reach the RCS Student Ambassador at <a href=\"mailto:rcs_sa_econ@scc.bu.edu\">rcs_sa_econ@scc.bu.edu<\/a><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p>Available software on the SCC: <a href=\" http:\/\/sccsvc.bu.edu\/software\/#\/\"> http:\/\/sccsvc.bu.edu\/software\/#\/<\/a><span style=\"font-weight: 400;\"><\/span><\/p>\n<p>The SCC and the pool of computers supporting the economics department are optimized for 28 core jobs. If you running large multi-core jobs (that is, using parallel processing), asking for 28 cores should get your jobs to start the fastest.<\/p>\n<p><b>Sample code<\/b><\/p>\n<p><span style=\"font-weight: 400;\">Code is also provided for the dynamic investment problem described in the slides. Examples in Gauss, Matlab, Python, Stata, and R are provided below. In some examples, separate code is given with and without parallel processing. Note that some problems are so simple that using parallel processing may slow the program down. The code is meant as an example. <\/span><\/p>\n<p><span style=\"font-weight: 400;\">Note that sites.bu.edu accepts files only with particular extensions, so the sample computer code has .txt extensions. You might want different extensions in practice. You need to remove the .txt from the batch file to use it.<\/span><\/p>\n<p><b>Gauss:<\/b><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\"><a href=\"https:\/\/www.bu.edu\/econ\/files\/2017\/07\/bellmanExample.txt\">Dynamic problem without parallel processing<\/a><\/span><\/li>\n<li><a href=\"https:\/\/www.bu.edu\/econ\/files\/2017\/07\/bellmanExampleMT.txt\"><span>Dynamic problem with parallel processing<\/span><\/a><span> (uses the ThreadStat command)<\/span><\/li>\n<\/ul>\n<p><b>Matlab:<\/b><\/p>\n<p><span style=\"font-weight: 400;\">Matlab has two ways of implementing parallel processing. Examples of both are provided. For this example, SPMD is a little more efficient.<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\"><a href=\"https:\/\/www.bu.edu\/econ\/files\/2017\/07\/BellmanMatlab.txt\">Dynamic problem without using parallel processing<\/a><\/span><\/li>\n<li><a href=\"https:\/\/www.bu.edu\/econ\/files\/2017\/07\/parallelBellmanMatlabParfor.txt\"><span>Dynamic problem with parallel processing using Parfor<\/span><\/a><\/li>\n<li><a href=\"https:\/\/www.bu.edu\/econ\/files\/2017\/07\/parallelBellmanMatlabSPMD.txt\"><span style=\"font-weight: 400;\">Dynamic problem with parallel processing using SPMD<\/span><\/a><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">You will also need these files to run in Matlab: <\/span><a href=\"https:\/\/www.bu.edu\/econ\/files\/2017\/07\/getVnew.txt\"><span style=\"font-weight: 400;\">getVnew<\/span><\/a> <a href=\"https:\/\/www.bu.edu\/econ\/files\/2017\/07\/profit.txt\"><span style=\"font-weight: 400;\">profit<\/span><\/a><\/p>\n<p><span style=\"font-weight: 400;\">Also, the slides describe a batch file for running Matlab in batch code on the cluster. Here is an example:<\/span><\/p>\n<p><a href=\"\/econ\/files\/2022\/02\/matlab_batch.txt\">Matlab_batch<\/a><br \/>\n<span style=\"font-weight: 400;\"> (Change the extension to .sh to use this batch file.)<\/span><\/p>\n<p><i><span style=\"font-weight: 400;\">Thanks goes to Mingli Chen and Kadin Tseng, who were a lot of help with the Matlab files.<\/span><\/i><\/p>\n<p><span style=\"font-weight: 400;\"><b>Please note to change the extension for batch files to .sh to use them for the below examples.<\/b><\/span><\/p>\n<p><strong>R:<\/strong><br \/>\nThis is an example of how to run a GMM estimation on R on clusters with a batch.<br \/>\n<a href=\"\/econ\/files\/2022\/02\/R_ReadMe.txt\">R_ReadMe<\/a> <a href=\"\/econ\/files\/2022\/02\/gmm_example.txt\">gmm_example<\/a> <a href=\"\/econ\/files\/2022\/02\/R_batch.txt\">R_batch<\/a><\/p>\n<p><strong>Stata:<\/strong><br \/>\nThis is an example of how to run a simple Stata do file on clusters with a batch.<br \/>\n<a href=\"\/econ\/files\/2022\/02\/R_ReadMe.txt\">R_ReadMe<\/a> <a href=\"\/econ\/files\/2022\/02\/do_example.txt\">do_example<\/a> <a href=\"\/econ\/files\/2022\/02\/Stata_batch.txt\">Stata_batch<\/a><\/p>\n<p><strong>Python:<\/strong><br \/>\nThis is an example of how to conduct web scrapping on Python on cluster with a batch.<br \/>\n<a href=\"\/econ\/files\/2022\/02\/Python_ReadMe.txt\">Python_ReadMe<\/a> <a href=\"\/econ\/files\/2022\/02\/webscrap.txt\">webscrap<\/a> <a href=\"\/econ\/files\/2022\/02\/Python_batch.txt\">Python_batch<\/a><\/p>\n<p><strong>Use array to more efficiently submit batch jobs.<\/strong><\/p>\n<p>Scenario: You have a number of R scripts to run, or want to run 1 script multiple times (e.g.<br \/>\nMaybe you\u2019re running simulations or doing a grid search over model parameters.)<\/p>\n<p>For example, for the R example above, you want to run a GMM estimation with four different sets of initial values. Instead of submit four batches, you can submit just one batch array.<\/p>\n<p><a href=\"\/econ\/files\/2022\/02\/gmm_example_1.txt\">gmm_example_1<\/a><a href=\"\/econ\/files\/2022\/02\/gmm_example_2.txt\">gmm_example_2<\/a> <a href=\"\/econ\/files\/2022\/02\/gmm_example_3.txt\">gmm_example_3<\/a> <a href=\"\/econ\/files\/2022\/02\/gmm_example_4.txt\">gmm_example_4<\/a> <a href=\"\/econ\/files\/2022\/02\/batch_array.txt\">batch_array<\/a><\/p>\n<p><a href=\"https:\/\/github.com\/bu-rcs\/SA-Economics\">https:\/\/github.com\/bu-rcs\/SA-Economics<\/a><\/p>\n<div class=\"bu_collapsible_container \" aria-live=\"polite\" data-customize-animation=\"false\"><h3 class=\"bu_collapsible\" aria-expanded=\"false\"tabindex=\"0\" role=\"button\"><b>Currently Available Data<\/b><\/h3><div class=\"bu_collapsible_section\" style=\"display: none;\"><\/p>\n<p><span style=\"font-weight: 400;\">The IED and the Department of Economics have purchased several licenses for various datasets to support students in their research.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">We are currently in the process of making more data available through the Research Data Network. Information regarding new data will be posted here as it become available.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Select the data below to open a detailed description:<\/span><\/p>\n<p><b>The Nielsen Datasets from the Kilts Center for Marketing<\/b><\/p>\n<p><b>About:<\/b><\/p>\n<p><span style=\"font-weight: 400;\">Approved users at Boston University can access the Retail Scanner data, the Consumer Panel data, and the PromoData. These datasets cover a wide range of products from a large set of retailers over time. The datasets are an ideal source for studying the purchase behavior of items typically purchased in grocery, drug, and convenience stores. The Retail Scanner data contains information on weekly price, sales, and store environment information provided by more than 90 retailers. The Consumer Panel data contains the purchases of fast-moving consumer goods for a set of 40,000 \u2013 60,000 households over time. The PromoData contains detailed manufacturer costs and allowances, introduction of new products, and price changes for all major grocery wholesalers from major markets.<\/span><\/p>\n<p><b>Obtaining Access:<\/b><\/p>\n<p><span style=\"font-weight: 400;\">Boston University has an institution-wide subscription to the Nielsen Datasets from Kilts. As such, the datasets are available to tenured faculty, tenure-track faculty, PhD students, and Post Doctorate students. To apply for access, follow the correct link below. Select \u201crequest subscriptions or register\u201d under the heading new users and find Boston University under the list of institutions. Then follow the instructions. The contact for the Nielsen Datasets is Adam Guren (guren@bu.edu).<\/span><\/p>\n<p><b>Links:<\/b><\/p>\n<p><span style=\"font-weight: 400;\">For more information and access:<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><a href=\"https:\/\/www.chicagobooth.edu\/research\/kilts\/datasets\/nielsen\"><span style=\"font-weight: 400;\">https:\/\/www.chicagobooth.edu\/research\/kilts\/datasets\/nielsen<\/span><\/a><\/p>\n<p><span style=\"font-weight: 400;\">To see how other researchers have used the Nielsen Datasets:<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><a href=\"https:\/\/papers.ssrn.com\/sol3\/JELJOUR_Results.cfm?form_name=journalbrowse&amp;journal_id=1829785\"><span style=\"font-weight: 400;\">https:\/\/papers.ssrn.com\/sol3\/JELJOUR_Results.cfm?form_name=journalbrowse&amp;journal_id=1829785<\/span><\/a><\/p>\n<p><b>Airline Origin and Destination Survey<\/b><\/p>\n<p><b>About:<\/b><\/p>\n<p><span style=\"font-weight: 400;\">The dataset consists of a 10% sample of an airline\u2019s tickets. The survey contains information on the origin, destination and intermediate points of flights, as well as, information on prices and distance travelled. It also contains information on the carriers, but does not include aircraft data.<\/span><\/p>\n<p><b>Access:<\/b><\/p>\n<p><span style=\"font-weight: 400;\">Information on domestic flights is publicly available but information on domestic to international flights is restricted. The Department of Transportation manages access to this restricted flight data. Marc Rysman has a copy of the restricted data and students can use it with Department of Transportation\u2019s approval. If you have questions regarding the data or obtaining access, please contact Marc Rysman at <\/span><a href=\"mailto:mrysman@bu.edu\"><span style=\"font-weight: 400;\">mrysman@bu.edu<\/span><\/a><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><b>Links:<\/b><\/p>\n<p><span style=\"font-weight: 400;\">For more information about the data,<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><a href=\"https:\/\/www.transtats.bts.gov\/tables.asp?Table_ID=272&amp;SYS_Table_Name=T_DB1B_TICKET\"><span style=\"font-weight: 400;\">https:\/\/www.transtats.bts.gov\/tables.asp?Table_ID=272&amp;SYS_Table_Name=T_DB1B_TICKET<\/span><\/a><\/p>\n<p><span style=\"font-weight: 400;\">For more information on obtaining Department of Transportation approval,<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><a href=\"https:\/\/www.bts.dot.gov\/topics\/airlines-and-airports\/restricted-data\"><span style=\"font-weight: 400;\">https:\/\/www.bts.dot.gov\/topics\/airlines-and-airports\/restricted-data<\/span><\/a><\/p>\n<p><b>Indian Firm Data<\/b><\/p>\n<p><b>About:<\/b><\/p>\n<p><span style=\"font-weight: 400;\">Available are data concerning firms in India<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Annual Survey of Industries (1998\/9 to 2011\/12)<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Economic Census (1998, 2005)<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">NSS Unorganized Manufacturing Surveys (2000\/1, 2005\/6, 2010\/11)<\/span><\/li>\n<\/ul>\n<p><b>Access:<\/b><\/p>\n<p><span style=\"font-weight: 400;\">For details regarding these and instructions for access, see <\/span><a href=\"https:\/\/www.bu.edu\/econ\/files\/2016\/01\/Presentation-on-BU-Indian-Firm-Data-April-3-2014-Aug-2015-update.pdf\"><span style=\"font-weight: 400;\">https:\/\/www.bu.edu\/econ\/files\/2016\/01\/Presentation-on-BU-Indian-Firm-Data-April-3-2014-Aug-2015-update.pdf<\/span><\/a><\/p>\n<p><\/div>\n<\/div>\n\n<div class=\"bu_collapsible_container \" aria-live=\"polite\" data-customize-animation=\"false\"><h3 class=\"bu_collapsible\" aria-expanded=\"false\"tabindex=\"0\" role=\"button\"><b>Data Acquired for PhD Research Projects<\/b><\/h3><div class=\"bu_collapsible_section\" style=\"display: none;\"><\/p>\n<p><b>About:<\/b><\/p>\n<p><span style=\"font-weight: 400;\">The following data was acquired for previous research projects, and has since become publicly available online.<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Firm-level data from the Annual Survey of Industries (ASI): 2000 \u2013 2013<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><a href=\"http:\/\/microdata.gov.in\/nada43\/index.php\/catalog\/ASI\"><span style=\"font-weight: 400;\">http:\/\/microdata.gov.in\/nada43\/index.php\/catalog\/ASI<\/span><\/a><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Unit-level data from the NSS Employment and Unemployment Survey: 1999 \u2013 2000<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><a href=\"http:\/\/microdata.gov.in\/nada43\/index.php\/catalog\/EUE\"><span style=\"font-weight: 400;\">http:\/\/microdata.gov.in\/nada43\/index.php\/catalog\/EUE<\/span><\/a><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Firm-level data from the NSS Unorganized Services: 2006 \u2013 2007<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><a href=\"http:\/\/microdata.gov.in\/nada43\/index.php\/catalog\/115\"><span style=\"font-weight: 400;\">http:\/\/microdata.gov.in\/nada43\/index.php\/catalog\/115<\/span><\/a><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">NSS Household Consumer Expenditure <\/span><a href=\"http:\/\/microdata.gov.in\/nada43\/index.php\/catalog\/CEXP\"><span style=\"font-weight: 400;\">http:\/\/microdata.gov.in\/nada43\/index.php\/catalog\/CEXP<\/span><\/a><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">The Sixth Economic Census offers a complete enumeration of all enterprises in India (except those engaged in crop plantation and cultivation, public administration, defense, and compulsory social security) and is identifiable at the village level. Available information includes but is not limited to the number of establishments, the number of persons employed therein, corresponding industries, and ownership status. Faculty and PhD students of the economics department interested in accessing this data should email <\/span><a href=\"mailto:iedcoord@bu.edu\"><span style=\"font-weight: 400;\">iedcoord@bu.edu<\/span><\/a><span style=\"font-weight: 400;\"> for more information.<\/span><\/p>\n<p><\/div>\n<\/div>\n\n","protected":false},"excerpt":{"rendered":"<p>Prof. Marc Rysman has given a seminar about why and how to use the cluster every few years. Here are the slides, with some minor updates: Prof. Schmieder&#8217;s Slides on Research Computing High Performance Computing for BU Economists (Previous version of slides) Important: The slides describe how to obtain access to the cluster. The faculty [&hellip;]<\/p>\n","protected":false},"author":15047,"featured_media":0,"parent":27003,"menu_order":5,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/www.bu.edu\/econ\/wp-json\/wp\/v2\/pages\/27063"}],"collection":[{"href":"https:\/\/www.bu.edu\/econ\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.bu.edu\/econ\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/econ\/wp-json\/wp\/v2\/users\/15047"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/econ\/wp-json\/wp\/v2\/comments?post=27063"}],"version-history":[{"count":32,"href":"https:\/\/www.bu.edu\/econ\/wp-json\/wp\/v2\/pages\/27063\/revisions"}],"predecessor-version":[{"id":37501,"href":"https:\/\/www.bu.edu\/econ\/wp-json\/wp\/v2\/pages\/27063\/revisions\/37501"}],"up":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/econ\/wp-json\/wp\/v2\/pages\/27003"}],"wp:attachment":[{"href":"https:\/\/www.bu.edu\/econ\/wp-json\/wp\/v2\/media?parent=27063"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}