Bug 205186 - USB ethernet device with NFS mount causes boot hang (startup order)
Summary: USB ethernet device with NFS mount causes boot hang (startup order)
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: usb (show other bugs)
Version: 10.2-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Hans Petter Selasky
URL:
Keywords: honeypot
Depends on: 205263
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-10 10:22 UTC by Brendan Molloy
Modified: 2024-04-17 18:18 UTC (History)
7 users (show)

See Also:


Attachments
console.log output (3.46 KB, text/plain)
2015-12-10 10:22 UTC, Brendan Molloy
no flags Details
Screen at time of hang (interlaced messages showing boot order issue) (255.63 KB, image/jpeg)
2015-12-10 10:27 UTC, Brendan Molloy
no flags Details
Updated diff for netwait script (5.37 KB, patch)
2015-12-17 23:50 UTC, Ian Lepore
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Brendan Molloy 2015-12-10 10:22:47 UTC
Created attachment 164034 [details]
console.log output

Today I've set up an old Inspiron 1525 laptop as a FreeBSD server. The internal NIC had died, so I am using an external NIC connected via USB. To my delight, FreeBSD detected it immediately and it just worked. It was wonderful!

However, troubles began when I added an NFS mount to my /etc/fstab. Upon booting, due to the fact the USB device isn't recognised and connected until after the booting process has finished, booting hangs with an error.

Dec 10 21:16:40 aerie kernel: Mounting NFS file systems:[tcp] 10.0.0.4:/nfs/BorgBackups: RPCPROG_NFS: RPC: Port mapper failure - RPC: Unable to send

Simply waiting, it will repeat this error after a while. I decided to try Ctrl+C, and booting completed. That caused this line in the console.log:

Dec 10 21:16:40 aerie kernel: Script /etc/rc.d/mountcritremote interrupted
Comment 1 Brendan Molloy 2015-12-10 10:27:05 UTC
Created attachment 164035 [details]
Screen at time of hang (interlaced messages showing boot order issue)
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2015-12-10 10:32:49 UTC
Thank you for the detailed report Brendan!
Comment 3 Hans Petter Selasky freebsd_committer freebsd_triage 2015-12-10 10:42:20 UTC
Hi,

Try MFC-ing r288265 to 10-stable.

https://svnweb.freebsd.org/changeset/base/288265

--HPS
Comment 4 Brendan Molloy 2015-12-10 10:48:59 UTC
I noticed a way to reproduce this issue while trying to fix some of my firewall rules.

Block all connections with a firewall (I used ipfilter). It will cause the hang at /etc/rc.d/mountcritremote when attempting to mount an NFS mount.

Once I fixed my rules, it still hangs for me on boot even with firewalls disabled, but eventually recovers when it retries after the network interface becomes available.
Comment 5 Ian Lepore freebsd_committer freebsd_triage 2015-12-10 15:31:06 UTC
(In reply to Hans Petter Selasky from comment #3)

I don't think r288265 will help -- it delays mounting root before init even begins, for diskless operation with an nfs root.  (I should still MFC it tho.)  Since the system isn't configured for nfsroot, it won't come into play.

In this case it's a matter of the network rc scripts running before the usb network interface is available, and I don't think we have any mechanism for waiting for NICs to arrive.

One thing that will work, but it's more of a workaround than a fix, is to set kern.cam.boot_delay=nnnnnn in loader.conf.  The delay is in milliseconds, so something like 10000 is probably enough.
Comment 6 Hans Petter Selasky freebsd_committer freebsd_triage 2015-12-10 15:34:29 UTC
OK, so is this an USB issue then?
Comment 7 Ian Lepore freebsd_committer freebsd_triage 2015-12-10 16:30:21 UTC
(In reply to Hans Petter Selasky from comment #6)

I don't think it's a problem in the sense of there being some usb driver code that can be changed to make it work -- it's an rc-scripts problem.

Oh, interesting... I've just discovered the existence of /etc/rc.d/netwait.  It looks like it's designed to handle exactly this situation... mountcritremote waits for netwait, and netwait can be configured to use a specific interface to ping an ip address and it doesn't complete until it gets a response or times out.  I think you just add netwait_enable=YES, netwait_if=<whatever>, netwait_ip=<ip to ping>.

If the interface is dhcp instead of static, just change it to SYNCDHCP in rc.conf
Comment 8 Brendan Molloy 2015-12-11 00:31:50 UTC
I tried the netwait method:

Dec 11 11:27:33 aerie kernel: Waiting for ue0 to have link
Dec 11 11:27:33 aerie kernel: /etc/rc: ERROR: ifconfig ue0 failed
Dec 11 11:27:33 aerie kernel: Mounting NFS file systems:[tcp] 10.0.0.4:/nfs/BorgBackups: RPCPROG_NFS: RPC: Port mapper failure - RPC: Unable to send

The interface does not exist at all until after the script had already run, so maybe the script should have an option to be more patient with errors?
Comment 9 Brendan Molloy 2015-12-11 14:09:28 UTC
Because I love FreeBSD, I am providing further information on this issue at 1am on a Saturday on Hans' and Kubilay's behalf.

I was asked to add hw.usb.axe.debug=16 to sysctl.conf, and I rebooted.

The only difference in logs was this:

Dec 12 01:01:34 aerie kernel: ue0: <USB Ethernet> on axe0
Dec 12 01:01:34 aerie kernel: ue0: Ethernet address: 00:50:b6:16:30:de
Dec 12 01:01:35 aerie ntpd[601]: ntpd 4.2.8p3-a (1): Starting
Dec 12 01:01:35 aerie kernel: .
Dec 12 01:01:36 aerie kernel: axe_bulk_write_callback: transfer complete
Dec 12 01:02:05 aerie last message repeated 10 times
Dec 12 01:02:53 aerie last message repeated 28 times
Comment 10 Ian Lepore freebsd_committer freebsd_triage 2015-12-11 16:22:22 UTC
(In reply to Brendan Molloy from comment #8)

Oh!  I should have realized... the interface doesn't exist until devd comes along and creates it, and that's too late.  I think you can fix that by adding to /boot/loader.conf:

  if_axe_load=YES

and you'll probably still need either SYNCDHCP or newait_enable for static ip.
Comment 11 Brendan Molloy 2015-12-12 05:27:25 UTC
(In reply to Ian Lepore from comment #10)

I tried this, and you are right, I still needed netwait.

The patch I provided in bug #205263 makes netwait more resilient to late loading of the modules, as it doesn't immediately scream about no interface when no iface is yet found. The timeout then gives enough time for the module to load at its own pace, while not requiring any changes to /boot/loader.conf.

If my patch is clean enough, it provides a solution inline with the principle of least astonishment to a problem that may become more prevalent in the future, as many modern laptops do not any longer include ethernet ports and users rely on USB or Thunderbolt ethernet adapters.
Comment 12 Ian Lepore freebsd_committer freebsd_triage 2015-12-17 23:50:45 UTC
Created attachment 164335 [details]
Updated diff for netwait script

I was updating the comments at the top of the script (seems to be the only "documentation" for netwait), and while describing how you could provide a list of interfaces to wait for, I realized it only allowed 1 interface.  I figured someone could have usb wifi and usb wired NIC or multiple NICs or whatever, so I updated the script to handle a list of interfaces.

I can only test the failure path (interface never arrives), it needs a test with real late-arriving hardware.
Comment 13 Ian Lepore freebsd_committer freebsd_triage 2015-12-18 01:04:11 UTC
I have turned the latest diff into a review at https://reviews.freebsd.org/D4608
Comment 14 commit-hook freebsd_committer freebsd_triage 2015-12-26 18:22:10 UTC
A commit references this bug:

Author: ian
Date: Sat Dec 26 18:21:33 UTC 2015
New revision: 292752
URL: https://svnweb.freebsd.org/changeset/base/292752

Log:
  Enhance rc.d/netwait script to wait for late-attaching interfaces such as
  USB NICs.

  USB network hardware may not be enumerated and available when the rc.d
  networking scripts run. Eventually the USB attachment completes and devd
  events cause the network initialization to happen, but by then other rc.d
  scripts have already failed, because services which depend on NETWORKING
  (such as mountcritremote) may end up running before the network is actually
  ready.

  There is an existing netwait script, but because it is dependent on
  NETWORKING it runs too late to prevent failure of some other rc
  scripts. This change flips the order so that NETWORKING depends on netwait,
  and netwait now depends on devd and routing (the former is needed to make
  interfaces appear, and the latter is needed to run the ping tests in
  netwait).

  The netwait script used to be oriented primarily towards "as soon as any
  host is reachable the network is fully functional", so you gave it a list of
  IPs to try and you could optionally name an interface and it would wait for
  carrier on that interface. That functionality still works the same, but now
  you can provide a list of interfaces to wait for and it waits until each one
  of them is available. The ping logic still completes as soon as the first IP
  on the list responds.

  These changes were submitted by Brenden Molloy <brendan+freebsd@bbqsrc.net>
  in PR 205186, and lightly modified by me to allow a list of interfaces
  instead of just one.

  PR:		205186
  Differential Revision:	https://reviews.freebsd.org/D4608 (timeout w/o review)

Changes:
  head/etc/defaults/rc.conf
  head/etc/rc.d/NETWORKING
  head/etc/rc.d/netwait
Comment 15 commit-hook freebsd_committer freebsd_triage 2016-01-24 19:41:36 UTC
A commit references this bug:

Author: ian
Date: Sun Jan 24 19:41:32 UTC 2016
New revision: 294680
URL: https://svnweb.freebsd.org/changeset/base/294680

Log:
  MFC r292752:

    Enhance rc.d/netwait script to wait for late-attaching interfaces such as
    USB NICs.

    USB network hardware may not be enumerated and available when the rc.d
    networking scripts run. Eventually the USB attachment completes and devd
    events cause the network initialization to happen, but by then other rc.d
    scripts have already failed, because services which depend on NETWORKING
    (such as mountcritremote) may end up running before the network is actually
    ready.

    There is an existing netwait script, but because it is dependent on
    NETWORKING it runs too late to prevent failure of some other rc
    scripts. This change flips the order so that NETWORKING depends on netwait,
    and netwait now depends on devd and routing (the former is needed to make
    interfaces appear, and the latter is needed to run the ping tests in
    netwait).

    The netwait script used to be oriented primarily towards "as soon as any
    host is reachable the network is fully functional", so you gave it a list of
    IPs to try and you could optionally name an interface and it would wait for
    carrier on that interface. That functionality still works the same, but now
    you can provide a list of interfaces to wait for and it waits until each one
    of them is available. The ping logic still completes as soon as the first IP
    on the list responds.

    These changes were submitted by Brenden Molloy <brendan+freebsd@bbqsrc.net>
    in PR 205186, and lightly modified by me to allow a list of interfaces
    instead of just one.

  PR:           205186
  Relnotes:	yes

Changes:
_U  stable/10/
  stable/10/etc/defaults/rc.conf
  stable/10/etc/rc.d/NETWORKING
  stable/10/etc/rc.d/netwait
Comment 16 neverendservices 2023-05-29 11:12:36 UTC
MARKED AS SPAM
Comment 17 Rockypatel 2023-07-09 14:54:20 UTC
MARKED AS SPAM
Comment 18 VN Editor For PC 2023-07-17 09:51:14 UTC
MARKED AS SPAM
Comment 19 ctbae 2023-08-08 06:16:32 UTC
MARKED AS SPAM
Comment 20 Dicik 2023-08-08 09:16:41 UTC
MARKED AS SPAM
Comment 21 streetgirls69 2023-08-08 09:25:18 UTC
MARKED AS SPAM
Comment 22 ctmot 2023-08-08 09:32:37 UTC
MARKED AS SPAM
Comment 23 depadas 2023-08-08 09:38:03 UTC
MARKED AS SPAM
Comment 24 Free Story 2023-09-06 07:10:52 UTC
MARKED AS SPAM
Comment 25 My Heaven Models Bangalore 2023-09-06 07:14:53 UTC
MARKED AS SPAM
Comment 26 My Heaven Models Bangalore 2023-09-06 07:16:33 UTC
MARKED AS SPAM
Comment 27 indiamyheavenmodels 2023-09-08 06:19:17 UTC
MARKED AS SPAM
Comment 28 Hyderabad Lily 2023-09-08 07:22:53 UTC
MARKED AS SPAM
Comment 29 priyanka 2023-09-08 08:14:55 UTC
MARKED AS SPAM
Comment 30 My Heaven Models Bangalore 2023-09-11 10:16:45 UTC
MARKED AS SPAM
Comment 31 visakhapatnam escorts 2023-09-11 13:25:51 UTC
MARKED AS SPAM
Comment 32 mussoorie myheavenmodels 2023-09-14 11:29:43 UTC
MARKED AS SPAM
Comment 33 priyanka 2023-09-16 07:35:34 UTC
MARKED AS SPAM
Comment 34 eatandrunpolice123 2023-09-20 10:09:00 UTC
MARKED AS SPAM
Comment 35 Kavya Gupta 2023-09-30 10:01:50 UTC
MARKED AS SPAM
Comment 36 Russian Escorts In Mumbai 2023-10-06 09:33:07 UTC
MARKED AS SPAM
Comment 37 Ethan carter 2023-10-11 05:49:04 UTC
MARKED AS SPAM
Comment 38 A Jaipur Escort 2023-10-12 12:41:20 UTC
MARKED AS SPAM
Comment 39 tinaduuta 2023-10-13 10:03:49 UTC
MARKED AS SPAM
Comment 40 tinaduuta 2023-10-13 10:04:34 UTC
MARKED AS SPAM
Comment 41 renudas 2023-10-17 09:48:00 UTC
MARKED AS SPAM
Comment 42 anna 2023-10-31 13:58:28 UTC
MARKED AS SPAM
Comment 43 Chhavi Garg 2023-12-07 05:41:17 UTC
MARKED AS SPAM
Comment 44 garimasuri 2023-12-08 05:45:06 UTC
MARKED AS SPAM
Comment 45 xedifo 2023-12-21 14:05:46 UTC
MARKED AS SPAM
Comment 46 Aashvi Khanna 2024-01-02 10:14:35 UTC
MARKED AS SPAM
Comment 47 isabasu 2024-01-08 10:03:32 UTC
MARKED AS SPAM
Comment 48 Neha Mari 2024-01-11 09:55:09 UTC
MARKED AS SPAM
Comment 49 Divya Aggarwal 2024-01-23 18:49:10 UTC
MARKED AS SPAM
Comment 50 Haridwar Escorts Dimpal 2024-01-29 15:56:58 UTC
MARKED AS SPAM
Comment 51 Haridwar Escorts Dimpal 2024-02-19 09:07:26 UTC
MARKED AS SPAM
Comment 52 India Escorts 2024-02-19 17:31:11 UTC
MARKED AS SPAM
Comment 53 India Escorts 2024-02-19 18:42:10 UTC
MARKED AS SPAM
Comment 54 India Escorts 2024-02-19 18:57:14 UTC
MARKED AS SPAM
Comment 55 India Escorts 2024-02-19 19:13:24 UTC
MARKED AS SPAM
Comment 56 agencymumbaigirl 2024-02-28 09:37:18 UTC
MARKED AS SPAM
Comment 57 riyamalik2u 2024-03-11 08:17:54 UTC
MARKED AS SPAM
Comment 58 Tina Kapoor 2024-03-12 06:37:59 UTC
MARKED AS SPAM
Comment 59 shruti goyal 2024-03-15 07:07:35 UTC
MARKED AS SPAM
Comment 60 shruti goyal 2024-03-15 07:08:10 UTC
MARKED AS SPAM
Comment 61 shruti goyal 2024-03-15 07:08:31 UTC
MARKED AS SPAM
Comment 62 shruti goyal 2024-03-15 07:08:49 UTC
MARKED AS SPAM
Comment 63 shruti goyal 2024-03-15 07:09:21 UTC
MARKED AS SPAM
Comment 64 shruti goyal 2024-03-15 07:10:26 UTC
MARKED AS SPAM
Comment 65 riyamalik2u 2024-03-15 07:10:53 UTC
MARKED AS SPAM
Comment 66 shruti goyal 2024-03-15 07:12:05 UTC
MARKED AS SPAM
Comment 67 vipmumbaiescorts 2024-04-06 04:58:34 UTC
Welcome to my site vipmumbaiescorts.net. I am an elegant Mumbai independent female , available for incall and outcall dates in Mumbai, as well as travel assignments vip Mumbai Escorts.
https://www.vipmumbaiescorts.net/
https://www.vipmumbaiescorts.net/location/andheri
Comment 69 Mumbai Escorts 2024-04-10 15:06:26 UTC
Mumbai escorts offering independent Call Girls Escort services in cheapest rates. Call us for best Escort services in mumbai for incall and outcall.
https://www.mumbaiescortshub.net

https://www.mumbaiescortshub.net/location/Andheri-call-girls.html
https://www.mumbaiescortshub.net/location/Vashi-call-girls.html
Comment 70 Tina Ji 2024-04-11 07:48:57 UTC
Faridabad Call Girls | Delhi Escorts | Mumbai Escorts | Pune Escorts | Gurgaon Escorts | Hyderabad Escorts | Chennai Escorts | Jaipur Escorts | Ludhiana Escorts | Chandigarh Escorts | Ahmedabad Escorts | Kolkata Escorts | Goa Escorts | Lucknow Escorts | Indore Escorts | Dehradun Escorts | Agartala Escorts | Pondicherry Escorts | Bhubaneswar Escorts | Dimapur Escorts | Aizawl Escorts | Shillong Escorts | Imphal Escorts | Bhopal Escorts | Kochi Escorts | Ranchi Escorts | Jammu Escorts | Shimla Escorts | Manali Escorts | Raipur Escorts | Patna Escorts | Guwahati Escorts | Visakhapatnam Escorts | Faridabad Escorts
Faridabad Escorts | Gurgaon Escorts | Hisar Escorts | Rohtak Escorts | Panipat Escorts | Karnal Escorts | Sonipat Escorts | Yamunanagar Escorts | Panchkula Escorts | Bhiwani Escorts | Bahadurgarh Escorts | Jind Escorts | Sirsa Escorts | Kaithal Escorts | Palwal Escorts | Rewari Escorts | Hansi Escorts | Narnaul Escorts | Fatehabad Escorts | Gohana Escorts | Tohana Escorts | Narwana Escorts | Mandi-Dabwali Escorts | Charkhi-Dadri Escorts | Samalkha Escorts | Pinjore Escorts | Ladwa Escorts | Safidon Escorts | Taraori Escorts | Mahendragarh Escorts | Ratia Escorts | Rania Escorts | Sarsod Escorts | Kurukshetra Escorts | Jhajjar Escorts
Call-Girls-In-Sector-3 Faridabad | Call-Girls-In Neharpar | Call-Girls-In-Green Fields | Call-Girls-In-Sainik Colony | Call-Girls-In-Old Faridabad | Call-Girls-In Palwal | Call-Girls-In-Ashoka Enclave | Call-Girls-In Ballabhgarh | Call-Girls-In-Sgm Nagar | Call-Girls-In-Suraj Kund | Call-Girls-In-Charmwood Village | Call-Girls-In-Dabuwa Colony | Call-Girls-In-Indraprastha Colony | Call-Girls-In-Surya Vihar | Call-Girls-In Badhkal | Call-Girls-In-Spring-Field Colony | Call-Girls-In Sehatpur | Call-Girls-In Kabulpur | Call-Girls-In Lakadpur | Call-Girls-In-Durga-Builder Colony | Call-Girls-In-Vinay Nagar | Call-Girls-In-Ajit Nagar | Call-Girls-In-Jawahar Colony | Call-Girls-In-Bhopani Village | Call-Girls-In Tigaon | Call-Girls-In Sholaka | Call-Girls-In-Chawla Colony | Call-Girls-In Agwanpur | Call-Girls-In-Bptp Parkland | Call-Girls-In-Ajronda Chowk | Call-Girls-In-Anangpur Dairy | Call-Girls-In Chandpur | Call-Girls-In-Friends-Colony Faridabad | Call-Girls-In-Rajeev-Nagar Faridabad | Call-Girls-In-Faridpur Village | Call-Girls-In-Manjhawali Village | Call-Girls-In Mithapur | Call-Girls-In-Hardware Colony | Call-Girls-In-Nehru Colony | Call-Girls-In Gazipur | Call-Girls-In Dhouj | Call-Girls-In-Bhatia Colony | Call-Girls-In-Industrial Area | Call-Girls-In-Neelam Chowk | Call-Girls-In-Shastri Colony | Call-Girls-In-Kanwara Village | Call-Girls-In Prithla | Call-Girls-In Tilpat | Call-Girls-In Ajronda | Call-Girls-In Ankhir | Call-Girls-In-Gopi Colony | Call-Girls-In-Gurukul Basti | Call-Girls-In Mujesar | Call-Girls-In-Yadav Colony | Call-Girls-In-Vishwakarma Colony | Call-Girls-In Shahedpur | Call-Girls-In-Basantpur Faridabad | Call-Girls-In-Titu Colony | Call-Girls-In-Tikawali Village | Call-Girls-In-Prithala Town | Call-Girls-In-Chand Hatt | Call-Girls-In-Rajpur Kalan | Call-Girls-In Jajru | Call-Girls-In Surajkund | Call-Girls-In Badkhal | Call-Girls-In Ballabgarh | Call-Girls-In-Greenfield Colony | Call-Girls-In-Green Field

Faridabad Call Girls | Call Girls in Faridabad | Faridabad Call Girl Service | Call Girl Service in Faridabad | Call Girl in Faridabad | Faridabad Call Girl
Escort in Aero city | Escort Chanakyapuri |Chhatarpur Escorts |Escort Connaught Place |Escort in Dwarka |Escort in Greater Kailash |Escort in Lajpat Nagar |Escort in Green Park |Hauz Khas Escorts |Escort in Janakpuri |Escort in Kalka Ji |Escort in Karol Bagh |Escort in Mahipalpur |Malviya Nagar Escorts |Mayur Vihar Escorts |Munirka Escorts |Escort in Nehru Place |Noida Escorts |Escort in Paharganj |Rajendra Place Escorts |Escort in Vasant Kunj |Saket Escorts |Vikaspuri Escorts |Rajouri Garden Escorts |Gurgaon Escorts |Surajkund Escorts |Vasant Kunj Escorts |Defence Colony Escorts | Friends Colony Escorts |Ghaziabad Escorts |Laxmi Nagar Escorts |Mehrauli Escorts |Mukherjee Nagar Escorts |South Ex Escorts |Ashok Nagar Escorts |Anand Vihar Escorts |Escorts in East of Kailash |Faridabad Call Girls |Delhi Call Girls |Mahipalpur Call Girls |Russian Escorts in Faridabad |Call Girl in Faridabad |Escorts Service in Faridabad |Female Escorts Surajkund |Housewife Escorts in Faridabad |Hi Profile Call Girls in Faridabad |Call Girls in Faridabad |Independent Call Girls in Faridabad
Comment 71 Tina Ji 2024-04-11 07:49:54 UTC
https://www.tinaji.com
https://www.tinaji.com/delhi-escorts.html
https://www.tinaji.com/mumbai-escorts.html
https://www.tinaji.com/pune-escorts.html
https://www.tinaji.com/gurgaon-escorts.html
https://www.tinaji.com/hyderabad-escorts.html
https://www.tinaji.com/chennai-escorts.html
https://www.tinaji.com/bangalore-escorts.html
https://www.tinaji.com/jaipur-escorts.html
https://www.tinaji.com/ludhiana-escorts.html
https://www.tinaji.com/chandigarh-escorts.html
https://www.tinaji.com/ahmedabad-escorts.html
https://www.tinaji.com/kolkata-escorts.html
https://www.tinaji.com/goa-escorts.html
https://www.tinaji.com/lucknow-escorts.html
https://www.tinaji.com/indore-escorts.html
https://www.tinaji.com/dehradun-escorts.html
https://www.tinaji.com/agartala-escorts.html
https://www.tinaji.com/pondicherry-escorts.html
https://www.tinaji.com/bhubaneswar-escorts.html
https://www.tinaji.com/dimapur-escorts.html
https://www.tinaji.com/aizawl-escorts.html
https://www.tinaji.com/shillong-escorts.html
https://www.tinaji.com/imphal-escorts.html
https://www.tinaji.com/bhopal-escorts.html
https://www.tinaji.com/kochi-escorts.html
https://www.tinaji.com/ranchi-escorts.html
https://www.tinaji.com/jammu-escorts.html
https://www.tinaji.com/shimla-escorts.html
https://www.tinaji.com/manali-escorts.html
https://www.tinaji.com/raipur-escorts.html
https://www.tinaji.com/patna-escorts.html
https://www.tinaji.com/guwahati-escorts.html
https://www.tinaji.com/visakhapatnam-escorts.html
https://www.tinakapoor.com/Faridabad-escorts.html

https://www.tinaji.com/faridabad-escorts.html
https://www.tinaji.com/gurgaon-escorts.html
https://www.tinaji.com/hisar-escorts.html
https://www.tinaji.com/rohtak-escorts.html
https://www.tinaji.com/panipat-escorts.html
https://www.tinaji.com/karnal-escorts.html
https://www.tinaji.com/sonipat-escorts.html
https://www.tinaji.com/yamunanagar-escorts.html
https://www.tinaji.com/panchkula-escorts.html
https://www.tinaji.com/bhiwani-escorts.html
https://www.tinaji.com/bahadurgarh-escorts.html
https://www.tinaji.com/jind-escorts.html
https://www.tinaji.com/sirsa-escorts.html
https://www.tinaji.com/kaithal-escorts.html
https://www.tinaji.com/palwal-escorts.html
https://www.tinaji.com/rewari-escorts.html
https://www.tinaji.com/hansi-escorts.html
https://www.tinaji.com/narnaul-escorts.html
https://www.tinaji.com/fatehabad-escorts.html
https://www.tinaji.com/gohana-escorts.html
https://www.tinaji.com/tohana-escorts.html
https://www.tinaji.com/narwana-escorts.html
https://www.tinaji.com/mandi-dabwali-escorts.html
https://www.tinaji.com/charkhi-dadri-escorts.html
https://www.tinaji.com/samalkha-escorts.html
https://www.tinaji.com/pinjore-escorts.html
https://www.tinaji.com/ladwa-escorts.html
https://www.tinaji.com/safidon-escorts.html
https://www.tinaji.com/taraori-escorts.html
https://www.tinaji.com/mahendragarh-escorts.html
https://www.tinaji.com/ratia-escorts.html
https://www.tinaji.com/rania-escorts.html
https://www.tinaji.com/sarsod-escorts.html
https://www.tinaji.com/kurukshetra-escorts.html
https://www.tinaji.com/jhajjar-escorts.html

https://www.tinaji.com/call-girls-in-sector-3-faridabad.html
https://www.tinaji.com/call-girls-in-neharpar.html
https://www.tinaji.com/call-girls-in-green-fields.html
https://www.tinaji.com/call-girls-in-sainik-colony.html
https://www.tinaji.com/call-girls-in-old-faridabad.html
https://www.tinaji.com/call-girls-in-palwal.html
https://www.tinaji.com/call-girls-in-ashoka-enclave.html
https://www.tinaji.com/call-girls-in-ballabhgarh.html
https://www.tinaji.com/call-girls-in-sgm-nagar.html
https://www.tinaji.com/call-girls-in-suraj-kund.html
https://www.tinaji.com/call-girls-in-charmwood-village.html
https://www.tinaji.com/call-girls-in-dabuwa-colony.html
https://www.tinaji.com/call-girls-in-indraprastha-colony.html
https://www.tinaji.com/call-girls-in-surya-vihar.html
https://www.tinaji.com/call-girls-in-badhkal.html
https://www.tinaji.com/call-girls-in-spring-field-colony.html
https://www.tinaji.com/call-girls-in-sehatpur.html
https://www.tinaji.com/call-girls-in-kabulpur.html
https://www.tinaji.com/call-girls-in-lakadpur.html
https://www.tinaji.com/call-girls-in-durga-builder-colony.html
https://www.tinaji.com/call-girls-in-vinay-nagar.html
https://www.tinaji.com/call-girls-in-ajit-nagar.html
https://www.tinaji.com/call-girls-in-jawahar-colony.html
https://www.tinaji.com/call-girls-in-bhopani-village.html
https://www.tinaji.com/call-girls-in-tigaon.html
https://www.tinaji.com/call-girls-in-sholaka.html
https://www.tinaji.com/call-girls-in-chawla-colony.html
https://www.tinaji.com/call-girls-in-agwanpur.html
https://www.tinaji.com/call-girls-in-bptp-parkland.html
https://www.tinaji.com/call-girls-in-ajronda-chowk.html
https://www.tinaji.com/call-girls-in-anangpur-dairy.html
https://www.tinaji.com/call-girls-in-chandpur.html
https://www.tinaji.com/call-girls-in-friends-colony-faridabad.html
https://www.tinaji.com/call-girls-in-rajeev-nagar-faridabad.html
https://www.tinaji.com/call-girls-in-faridpur-village.html
https://www.tinaji.com/call-girls-in-manjhawali-village.html
https://www.tinaji.com/call-girls-in-mithapur.html
https://www.tinaji.com/call-girls-in-hardware-colony.html
https://www.tinaji.com/call-girls-in-nehru-colony.html
https://www.tinaji.com/call-girls-in-gazipur.html
https://www.tinaji.com/call-girls-in-dhouj.html
https://www.tinaji.com/call-girls-in-bhatia-colony.html
https://www.tinaji.com/call-girls-in-industrial-area.html
https://www.tinaji.com/call-girls-in-neelam-chowk.html
https://www.tinaji.com/call-girls-in-shastri-colony.html
https://www.tinaji.com/call-girls-in-kanwara-village.html
https://www.tinaji.com/call-girls-in-prithla.html
https://www.tinaji.com/call-girls-in-tilpat.html
https://www.tinaji.com/call-girls-in-ajronda.html
https://www.tinaji.com/call-girls-in-ankhir.html
https://www.tinaji.com/call-girls-in-gopi-colony.html
https://www.tinaji.com/call-girls-in-gurukul-basti.html
https://www.tinaji.com/call-girls-in-mujesar.html
https://www.tinaji.com/call-girls-in-yadav-colony.html
https://www.tinaji.com/call-girls-in-vishwakarma-colony.html
https://www.tinaji.com/call-girls-in-shahedpur.html
https://www.tinaji.com/call-girls-in-basantpur-faridabad.html
https://www.tinaji.com/call-girls-in-titu-colony.html
https://www.tinaji.com/call-girls-in-tikawali-village.html
https://www.tinaji.com/call-girls-in-prithala-town.html
https://www.tinaji.com/call-girls-in-chand-hatt.html
https://www.tinaji.com/call-girls-in-rajpur-kalan.html
https://www.tinaji.com/call-girls-in-jajru.html
https://www.tinaji.com/call-girls-in-surajkund.html
https://www.tinaji.com/call-girls-in-badkhal.html
https://www.tinaji.com/call-girls-in-ballabgarh.html
https://www.tinaji.com/call-girls-in-greenfield-colony.html
https://www.tinaji.com/call-girls-in-green-field.html

https://www.callgirlsinfaridabad.com/
https://www.callgirlsinfaridabad.com/escort-in-aerocity.html
https://www.callgirlsinfaridabad.com/escort-chanakyapuri.html
https://www.callgirlsinfaridabad.com/chhatarpur-escorts.html
https://www.callgirlsinfaridabad.com/escort-connaught-place.html
https://www.callgirlsinfaridabad.com/escort-in-dwarka.html
https://www.callgirlsinfaridabad.com/escort-in-greater-kailash.html
https://www.callgirlsinfaridabad.com/escort-in-lajpat-nagar.html
https://www.callgirlsinfaridabad.com/escort-in-green-park.html
https://www.callgirlsinfaridabad.com/hauz-khas-escorts.html
https://www.callgirlsinfaridabad.com/escort-in-janakpuri.html
https://www.callgirlsinfaridabad.com/escort-in-kalkaji.html
https://www.callgirlsinfaridabad.com/escort-in-karol-bagh.html
https://www.callgirlsinfaridabad.com/escort-in-mahipalpur.html
https://www.callgirlsinfaridabad.com/malviya-nagar-escorts.html
https://www.callgirlsinfaridabad.com/mayur-vihar-escorts.html
https://www.callgirlsinfaridabad.com/munirka-escorts.html
https://www.callgirlsinfaridabad.com/escort-in-nehru-place.html
https://www.callgirlsinfaridabad.com/noida-escorts.html
https://www.callgirlsinfaridabad.com/escort-in-paharganj.html
https://www.callgirlsinfaridabad.com/rajendra-place-escorts.html
https://www.callgirlsinfaridabad.com/escort-in-vasant-kunj.html
https://www.callgirlsinfaridabad.com/saket-escorts.html
https://www.callgirlsinfaridabad.com/vikaspuri-escorts.html
https://www.callgirlsinfaridabad.com/rajouri-garden-escorts.html
https://www.callgirlsinfaridabad.com/gurgaon-escorts.html
https://www.callgirlsinfaridabad.com/surajkund-escorts.html
https://www.callgirlsinfaridabad.com/vasant-kunj-escorts.html
https://www.callgirlsinfaridabad.com/defence-colony-escorts.html
https://www.callgirlsinfaridabad.com/friends-colony-escorts.html
https://www.callgirlsinfaridabad.com/ghaziabad-escorts.html
https://www.callgirlsinfaridabad.com/laxmi-nagar-escorts.html
https://www.callgirlsinfaridabad.com/mehrauli-escorts.html
https://www.callgirlsinfaridabad.com/mukherji-nagar-escorts.html
https://www.callgirlsinfaridabad.com/south-ex-escorts.html
https://www.callgirlsinfaridabad.com/ashok-nagar-escorts.html
https://www.callgirlsinfaridabad.com/anand-vihar-escorts.html
https://www.callgirlsinfaridabad.com/escorts-in-east-of-kailash.html

https://www.callgirlsinfaridabad.com/about-us.html
https://www.callgirlsinfaridabad.com/delhi-call-girls.html
https://www.callgirlsinfaridabad.com/mahipalpur-call-girls.html
https://www.callgirlsinfaridabad.com/russian-escorts-in-faridabad.html
https://www.callgirlsinfaridabad.com/contact-us.html
https://www.callgirlsinfaridabad.com/escorts-service-in-faridabad.html
https://www.callgirlsinfaridabad.com/female-escorts-surujkund.html
https://www.callgirlsinfaridabad.com/housewife-escorts-in-faridabad.html
https://www.callgirlsinfaridabad.com/hi-profile-call-girls-in-faridabad.html
https://www.callgirlsinfaridabad.com/gallery.html
https://www.callgirlsinfaridabad.com/independent-call-girls-in-faridabad.html
Comment 72 Monaroy 2024-04-17 18:18:44 UTC
Call Girl Jaipur at a cheap rate of 2500 with A/C Hotel Room. InCall & OutCall Facility available cash on delivery within 25 minutes Only.
https://escortsjaipur.com/