加入 Gitee

与超过 1200万开发者一起发现、参与优秀开源项目，私有仓库也完全免费：）

克隆/下载

data-science.md 7.58 KB

# Data Science

**Table of Contents**

- [Doc](#doc)
- [Open data](#open-data)
- [Data storage](#data-storage)
- [Markup Language](#markup-language)
- [Languages](#languages)
- [Workflow/Pipelines tools](#workflowpipelines-tools)
- [Dataset](#dataset)
- [Tools](#tools)
- [Data structure](#data-structure)
- [Algorithm](#algorithm)
- [Statistics](#statistics)
- [Big Data and Cloud](#big-data-and-cloud)
- [Books](#books)
- [Course](#course)
- [Misc](#misc)

## Doc

-  [Three Things About Data Science You Won't Find In the Books](http://blog.mikiobraun.de/2015/03/three-things-about-data-science.html)

## Open data

- [awesome-public-datasets](https://github.com/awesomedata/awesome-public-datasets) - A topic-centric list of high-quality open datasets in public domains. By everyone, for everyone!
- [fivethirtyeight/data](https://github.com/fivethirtyeight/data) - Data and code behind the articles and graphics at FiveThirtyEight https://data.fivethirtyeight.com/

## Data storage

- [Ten Simple Rulesfor Digital Data Storage](http://journals.plos.org/ploscompbiol/article/asset?id=10.1371/journal.pcbi.1005097.PDF)
- [How to share data with a statistician](https://github.com/jtleek/datasharing)

New tech

- [IPFS](https://ipfs.io/) is the Distributed Web

## Markup Language

- [YAML 语言教程](http://www.ruanyifeng.com/blog/2016/07/yaml.html)

## Languages

-  [Data Science with Python & R: Dimensionality Reduction and Clustering](https://www.codementor.io/python/tutorial/data-science-python-pandas-r-dimensionality-reduction)
-  [R vs Python: head to head data analysis](https://www.dataquest.io/blog/python-vs-r/) | [R vs Python：硬碰硬的数据分析](http://mp.weixin.qq.com/s?__biz=MzA4OTg5NzY3NA==&mid=208986906&idx=4&sn=799e06b3d09acb8025df550aca3596a9)
-  [DataPyR](https://datapyr.zeef.com/kranthi.kumar)
-  [Choosing R or Python for data analysis? An infographic](http://blog.datacamp.com/r-or-python-for-data-analysis/)

## Workflow/Pipelines tools

DSL

-  [GNU make](https://www.gnu.org/software/make/), [manual](https://www.gnu.org/software/make/manual/), [Make 命令教程](http://www.ruanyifeng.com/blog/2015/02/make.html)
-  [Common Workflow Language](http://www.commonwl.org/)
-  [***snakemake***](https://bitbucket.org/snakemake/snakemake/wiki/Home) - Snakemake is a workflow management system that aims to reduce the complexity of creating workflows by providing a fast and comfortable execution environment, together with a clean and modern specification language in python style.  [Build bioinformatics pipelines with Snakemake](https://slowkow.com/notes/snakemake-tutorial/)
-  [**nextflow**](https://github.com/nextflow-io/nextflow) - A DSL for data-driven computational pipelines http://nextflow.io
-  [sake](https://github.com/tonyfischetti/sake) - A self-documenting build automation tool

Language-dependent

-  [toil](https://github.com/BD2KGenomics/toil) - A scalable, efficient, cross-platform and easy-to-use workflow engine in pure Python
-  [Ruffus](http://www.ruffus.org.uk/) - Ruffus is a Computation Pipeline library for python. It is open-sourced, powerful and user-friendly, and widely used in science and bioinformatics.

## Dataset

-  [awesome-public-datasets](https://github.com/caesar0301/awesome-public-datasets) - An awesome list of (large-scale) public datasets on the Internet. (On-going collection)

## Tools

-  [csvkit](https://github.com/onyxfish/csvkit) - A suite of utilities for converting to and working with CSV, the king of tabular file formats. [http://csvkit.rtfd.org/](http://csvkit.rtfd.org/)
-  [csvtk](https://github.com/shenwei356/csvtk) - Another cross-platform, efficient, practical and pretty CSV/TSV toolkit in Golang http://shenwei356.github.io/csvtk
-  [icdiff](https://github.com/jeffkaufman/icdiff) - improved colored diff http://www.jefftk.com/icdiff

## Data structure

-  [skizze](https://github.com/seiflotfy/skizze) - A probabilistic data structure service and storage
-  [dablooms](https://github.com/bitly/dablooms) - A Scalable, Counting, Bloom Filter. Java, Python, Go edition.
-  [inbloom](https://github.com/EverythingMe/inbloom) - Cross language bloom filter implementation
-  HyperLogLog & HyperLogLog++,
   [大数据计算：如何仅用1.5KB内存为十亿对象计数 - Hyper LogLog 算法](http://blog.csdn.net/heiyeshuwu/article/details/41248379),
   [如何快速估计巨大 dataset 中unique 元素的数目](http://baozitraining.org/blog/how-to-quickly-count-unique-items-in-large-dataset/)

## Algorithm

-  [Fast and Easy Levenshtein distance using a Trie](http://stevehanov.ca/blog/index.php?id=114)

## Statistics

-  [YGC的统计笔记](http://ygc.name/stats/)
-  [《On the scalability of statistical procedures: why the p-value bashers just don't get it》](http://simplystatistics.org/2014/02/14/on-the-scalability-of-statistical-procedures-why-the-p-value-bashers-just-dont-get-it/)by [Jeff Leek](http://simplystatistics.org/author/jtleek/)
-  [The Only Probability Cheatsheet You'll Ever Need](http://www.wzchen.com/probability-cheatsheet/)
-  [Where priors come from](http://zinkov.com/posts/2015-06-09-where-priors-come-from/) (some popular distribution)
-  [Probabilistic Programming & Bayesian Methods for Hackers](https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers)
-  [The Advanced Matrix Factorization Jungle](https://sites.google.com/site/igorcarron2/matrixfactorizations)
-  [R, STATISTICS, PSYCHOLOGY, OPEN SCIENCE, DATA VISUALIZATION](http://rpsychologist.com/)

p-value

- [Scientific method: Statistical errors](http://www.nature.com/news/scientific-method-statistical-errors-1.14700)
- [P value ban: small step for a journal, giant leap for science](https://www.sciencenews.org/blog/context/p-value-ban-small-step-journal-giant-leap-science)
- [Science Isn’t Broken](http://fivethirtyeight.com/features/science-isnt-broken/)
- [Odds Are, It's Wrong](https://www.sciencenews.org/article/odds-are-its-wrong)

## Big Data and Cloud

-  [左耳朵耗子谈云计算：拼的就是运维](http://www.infoq.com/cn/articles/chenhao-on-cloud)
-  [分布式系统的事务处理](http://coolshell.cn/articles/10910.html)
-  [【专治不明觉厉】之“大数据”](http://www.huxiu.com/article/31457/1.html?f=zaker)
-  [hadoop和大数据的关系？和spark的关系？互补？并行？](http://www.zhihu.com/question/23036370)
-  [后Hadoop时代的大数据架构](http://www.36dsj.com/archives/28450)
-  [Elements of Scale: Composing and Scaling Data Platforms](http://www.benstopford.com/2015/04/28/elements-of-scale-composing-and-scaling-data-platforms/)
-  http://www.analyticsvidhya.com/
-  [数据科学家修炼宝典](http://mooc.guokr.com/topic/DataScientist/)

## Books

- [***Mining of Massive Datasets***](http://www.mmds.org/) - The book is based on Stanford Computer Science course CS246: Mining Massive Datasets

## Course

- [Stanford Computer Science course CS246: Mining Massive Datasets](http://www.ee.columbia.edu/~cylin/course/bigdata/index.html)
- [algorithms for Big Data](http://grigory.us/big-data-class.html) - This class will give you a biased sample of techniques for scalable data anslysis

## Misc

-  [What to do with “small” data?](https://medium.com/rants-on-machine-learning/what-to-do-with-small-data-d253254d1a89#.720upf8f1)
-  [Convert xlsx to csv in linux command line](http://stackoverflow.com/questions/10557360/convert-xlsx-to-csv-in-linux-command-line), use `ssconvert` of Gnumeric

一键复制编辑原始数据按行查看历史

提交于 2018-03-07 00:05 . 2018-03-06

Data Science

Table of Contents

Doc
Open data
Data storage
Markup Language
Languages
Workflow/Pipelines tools
Dataset
Tools
Data structure
Algorithm
Statistics
Big Data and Cloud
Books
Course
Misc

Doc

Three Things About Data Science You Won't Find In the Books

Open data

awesome-public-datasets - A topic-centric list of high-quality open datasets in public domains. By everyone, for everyone!
fivethirtyeight/data - Data and code behind the articles and graphics at FiveThirtyEight https://data.fivethirtyeight.com/

Data storage

New tech

IPFS is the Distributed Web

Markup Language

YAML 语言教程

Languages

Workflow/Pipelines tools

DSL

GNU make, manual, Make 命令教程
Common Workflow Language
snakemake - Snakemake is a workflow management system that aims to reduce the complexity of creating workflows by providing a fast and comfortable execution environment, together with a clean and modern specification language in python style. Build bioinformatics pipelines with Snakemake
nextflow - A DSL for data-driven computational pipelines http://nextflow.io
sake - A self-documenting build automation tool

Language-dependent

toil - A scalable, efficient, cross-platform and easy-to-use workflow engine in pure Python
Ruffus - Ruffus is a Computation Pipeline library for python. It is open-sourced, powerful and user-friendly, and widely used in science and bioinformatics.

Dataset

awesome-public-datasets - An awesome list of (large-scale) public datasets on the Internet. (On-going collection)

Tools

csvkit - A suite of utilities for converting to and working with CSV, the king of tabular file formats. http://csvkit.rtfd.org/
csvtk - Another cross-platform, efficient, practical and pretty CSV/TSV toolkit in Golang http://shenwei356.github.io/csvtk
icdiff - improved colored diff http://www.jefftk.com/icdiff

Data structure

skizze - A probabilistic data structure service and storage
dablooms - A Scalable, Counting, Bloom Filter. Java, Python, Go edition.
inbloom - Cross language bloom filter implementation
HyperLogLog & HyperLogLog++, 大数据计算：如何仅用1.5KB内存为十亿对象计数 - Hyper LogLog 算法, 如何快速估计巨大 dataset 中unique 元素的数目

Algorithm

Fast and Easy Levenshtein distance using a Trie

Statistics

p-value

Big Data and Cloud

Books

Mining of Massive Datasets - The book is based on Stanford Computer Science course CS246: Mining Massive Datasets

Course

Stanford Computer Science course CS246: Mining Massive Datasets
algorithms for Big Data - This class will give you a biased sample of techniques for scalable data anslysis

Misc

What to do with “small” data?
Convert xlsx to csv in linux command line, use ssconvert of Gnumeric

1

https://gitee.com/ShixiangWang/awesome.git

git@gitee.com:ShixiangWang/awesome.git

ShixiangWang

awesome

awesome

master