1 Star 1 Fork 234

哲涵1111 / QueryList

forked from JAE / QueryList 
Create your Gitee Account
Explore and code with more than 6 million developers,Free private repositories !:)
Sign up
This repository doesn't specify license. Without author's permission, this code is only for learning and cannot be used for other purposes.
Clone or download
README.md 9.08 KB
Copy Edit Web IDE Raw Blame History
jaegercode authored 2017-10-09 11:27 . ok



QueryList is a simple, elegant, extensible PHP Web Scraper (crawler/spider) ,based on phpQuery.

API Documentation



  • Have the same CSS3 DOM selector as jQuery
  • Have the same DOM manipulation API as jQuery
  • Have a generic list crawling program
  • Have a strong HTTP request suite, easy to achieve such as: simulated landing, forged browser, HTTP proxy and other complex network requests
  • Have a messy code solution
  • Have powerful content filtering, you can use the jQuey selector to filter content
  • Has a high degree of modular design, scalability and strong
  • Have an expressive API
  • Has a wealth of plug-ins

Through plug-ins you can easily implement things like:

  • Multithreaded crawl
  • Crawl JavaScript dynamic rendering page (PhantomJS/headless WebKit)
  • Image downloads to local
  • Simulate browser behavior such as submitting Form forms
  • Web crawler
  • .....


  • PHP >= 7.0


By Composer installation:

composer require jaeger/querylist


DOM Traversal and Manipulation

  • Crawl「GitHub」all picture links
  • Crawl Google search results
$ql = QueryList::get('https://www.google.co.jp/search?q=QueryList');

$ql->find('title')->text(); //The page title
$ql->find('meta[name=keywords]')->content; //The page keywords

$ql->find('h3>a')->texts(); //Get a list of search results titles
$ql->find('h3>a')->attrs('href'); //Get a list of search results links

$ql->find('img')->src; //Gets the link address of the first image
$ql->find('img:eq(1)')->src; //Gets the link address of the second image
$ql->find('img')->eq(2)->src; //Gets the link address of the third image
// Loop all the images
	echo $img->alt;  //Print the alt attribute of the image
  • More usage
$ql->find('#head')->append('<div>Append content</div>')->find('div')->htmls();
$ql->find('.two')->children('img')->attrs('alt'); // Get the class is the "two" element under all img child nodes
// Loop class is the "two" element under all child nodes
$data = $ql->find('.two')->children()->map(function ($item){
    // Use "is" to determine the node type
        return $item->text();
        return $item->alt;

$ql->find('a')->attr('href', 'newVal')->removeClass('className')->html('newHtml')->...
$ql->find('div > p')->add('div > ul')->filter(':has(a)')->find('p:first')->nextAll()->andSelf()->...
$ql->find('div.old')->replaceWith( $ql->find('div.new')->clone())->appendTo('.trash')->prepend('Deleted')->...

List crawl

Crawl the title and link of the Google search results list:

$data = QueryList::get('https://www.google.co.jp/search?q=QueryList')
	// Set the crawl rules



    [0] => Array
            [title] => Angular - QueryList
            [link] => https://angular.io/api/core/QueryList
    [1] => Array
            [title] => QueryList | @angular/core - Angularリファレンス - Web Creative Park
            [link] => http://www.webcreativepark.net/angular/querylist/
    [2] => Array
            [title] => QueryListにQueryを追加したり、追加されたことを感知する | TIPS ...
            [link] => http://www.webcreativepark.net/angular/querylist_query_add_subscribe/

Encode convert

// Out charset :UTF-8
// In charset :GB2312

// Out charset:UTF-8
// In charset:Automatic Identification

HTTP Client (GuzzleHttp)

  • Carry cookie login GitHub
//Crawl GitHub content
$ql = QueryList::get('https://github.com','param1=testvalue & params2=somevalue',[
  'headers' => [
      // Fill in the cookie from the browser
      'Cookie' => 'SINAGLOBAL=546064; wb_cmtLike_2112031=1; wvr=6;....'
//echo $ql->getHtml();
$userName = $ql->find('.header-nav-current-user>.css-truncate-target')->text();
echo $userName;
  • Use the Http proxy
$urlParams = ['param1' => 'testvalue','params2' => 'somevalue'];
$opts = [
	// Set the http proxy
    'proxy' => '',
    //Set the timeout time in seconds
    'timeout' => 30,
     // Fake HTTP headers
    'headers' => [
        'Referer' => 'https://querylist.cc/',
        'User-Agent' => 'testing/1.0',
        'Accept'     => 'application/json',
        'X-Foo'      => ['Bar', 'Baz'],
        'Cookie'    => 'abc=111;xxx=222'
// echo $ql->getHtml();
  • Analog login
// Post login
$ql = QueryList::post('http://xxxx.com/login',[
    'username' => 'admin',
    'password' => '123456'
// Crawl pages that need to be logged in to access
//echo $ql->getHtml();

Submit forms

Login GitHub

// Get the QueryList instance
$ql = QueryList::getInstance();
// Get the login form
$form = $ql->get('https://github.com/login')->find('form');

// Fill in the GitHub username and password
$form->find('input[name=login]')->val('your github username or email');
$form->find('input[name=password]')->val('your github password');

// Serialize the form data
$fromData = $form->serializeArray();
$postData = [];
foreach ($fromData as $item) {
    $postData[$item['name']] = $item['value'];

// Submit the login form
$actionUrl = 'https://github.com'.$form->attr('action');
// To determine whether the login is successful
// echo $ql->getHtml();
$userName = $ql->find('.header-nav-current-user>.css-truncate-target')->text();
    echo 'Login successful ! Welcome:'.$userName;
    echo 'Login failed !';

Bind function extension

Customize the extension of a myHttp method:

$ql = QueryList::getInstance();

//Bind a `myHttp` method to the QueryList object
$ql->bind('myHttp',function ($url){
	// $this is the current QueryList object
    $html = file_get_contents($url);
    return $this;

// And then you can call by the name of the binding
$data = $ql->myHttp('https://toutiao.io')->find('h3 a')->texts();

Or package to class, and then bind:

$ql->bind('myHttp',function ($url){
    return new MyHttp($this,$url);

Plugin used

  • Use the PhantomJS plugin to crawl JavaScript dynamically rendered pages:
// Set the PhantomJS binary file path during installation
$ql = QueryList::use(PhantomJs::class,'/usr/local/bin/phantomjs');

// Crawl「500px」all picture links
$data = $ql->browser('https://500px.com/editors')->find('img')->attrs('src');

// Use the HTTP proxy
	'--proxy' => '',
    '--proxy-type' => 'http'
  • Using the CURL multithreading plug-in, multi-threaded crawling GitHub trending :
$ql = QueryList::use(CurlMulti::class);
    //.....more urls
 // Called if task is success
 ->success(function (QueryList $ql,CurlMulti $curl,$r){
    echo "Current url:{$r['info']['url']} \r\n";
    $data = $ql->find('h3 a')->texts();
 // Task fail callback
->error(function ($errorInfo,CurlMulti $curl){
    echo "Current url:{$errorInfo['info']['url']} \r\n";
	// Maximum number of threads
    'maxThread' => 10,
    // Number of error retries
    'maxTry' => 3,


View more QueryList plugins and QueryList-based products: QueryList Community


Welcome to contribute code for the QueryList。About Contributing Plugins can be viewed:QueryList Plugin Contributing Guide


Jaeger JaegerCode@gmail.com

If this library is useful for you, say thanks buying me a beer :beer:!


QueryList is licensed under the license of MIT. See the LICENSE for more details.

Comment ( 0 )

Sign in for post a comment



010926 d3736ce6 1850385 010927 f77ebe53 1850385