Bug 205186 - USB ethernet device with NFS mount causes boot hang (startup order)
Summary: USB ethernet device with NFS mount causes boot hang (startup order)
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: usb (show other bugs)
Version: 10.2-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Hans Petter Selasky
URL:
Keywords: honeypot
Depends on: 205263
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-10 10:22 UTC by Brendan Molloy
Modified: 2024-07-01 12:02 UTC (History)
20 users (show)

See Also:


Attachments
console.log output (3.46 KB, text/plain)
2015-12-10 10:22 UTC, Brendan Molloy
no flags Details
Screen at time of hang (interlaced messages showing boot order issue) (255.63 KB, image/jpeg)
2015-12-10 10:27 UTC, Brendan Molloy
no flags Details
Updated diff for netwait script (5.37 KB, patch)
2015-12-17 23:50 UTC, Ian Lepore
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Brendan Molloy 2015-12-10 10:22:47 UTC
Created attachment 164034 [details]
console.log output

Today I've set up an old Inspiron 1525 laptop as a FreeBSD server. The internal NIC had died, so I am using an external NIC connected via USB. To my delight, FreeBSD detected it immediately and it just worked. It was wonderful!

However, troubles began when I added an NFS mount to my /etc/fstab. Upon booting, due to the fact the USB device isn't recognised and connected until after the booting process has finished, booting hangs with an error.

Dec 10 21:16:40 aerie kernel: Mounting NFS file systems:[tcp] 10.0.0.4:/nfs/BorgBackups: RPCPROG_NFS: RPC: Port mapper failure - RPC: Unable to send

Simply waiting, it will repeat this error after a while. I decided to try Ctrl+C, and booting completed. That caused this line in the console.log:

Dec 10 21:16:40 aerie kernel: Script /etc/rc.d/mountcritremote interrupted
Comment 1 Brendan Molloy 2015-12-10 10:27:05 UTC
Created attachment 164035 [details]
Screen at time of hang (interlaced messages showing boot order issue)
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2015-12-10 10:32:49 UTC
Thank you for the detailed report Brendan!
Comment 3 Hans Petter Selasky freebsd_committer freebsd_triage 2015-12-10 10:42:20 UTC
Hi,

Try MFC-ing r288265 to 10-stable.

https://svnweb.freebsd.org/changeset/base/288265

--HPS
Comment 4 Brendan Molloy 2015-12-10 10:48:59 UTC
I noticed a way to reproduce this issue while trying to fix some of my firewall rules.

Block all connections with a firewall (I used ipfilter). It will cause the hang at /etc/rc.d/mountcritremote when attempting to mount an NFS mount.

Once I fixed my rules, it still hangs for me on boot even with firewalls disabled, but eventually recovers when it retries after the network interface becomes available.
Comment 5 Ian Lepore freebsd_committer freebsd_triage 2015-12-10 15:31:06 UTC
(In reply to Hans Petter Selasky from comment #3)

I don't think r288265 will help -- it delays mounting root before init even begins, for diskless operation with an nfs root.  (I should still MFC it tho.)  Since the system isn't configured for nfsroot, it won't come into play.

In this case it's a matter of the network rc scripts running before the usb network interface is available, and I don't think we have any mechanism for waiting for NICs to arrive.

One thing that will work, but it's more of a workaround than a fix, is to set kern.cam.boot_delay=nnnnnn in loader.conf.  The delay is in milliseconds, so something like 10000 is probably enough.
Comment 6 Hans Petter Selasky freebsd_committer freebsd_triage 2015-12-10 15:34:29 UTC
OK, so is this an USB issue then?
Comment 7 Ian Lepore freebsd_committer freebsd_triage 2015-12-10 16:30:21 UTC
(In reply to Hans Petter Selasky from comment #6)

I don't think it's a problem in the sense of there being some usb driver code that can be changed to make it work -- it's an rc-scripts problem.

Oh, interesting... I've just discovered the existence of /etc/rc.d/netwait.  It looks like it's designed to handle exactly this situation... mountcritremote waits for netwait, and netwait can be configured to use a specific interface to ping an ip address and it doesn't complete until it gets a response or times out.  I think you just add netwait_enable=YES, netwait_if=<whatever>, netwait_ip=<ip to ping>.

If the interface is dhcp instead of static, just change it to SYNCDHCP in rc.conf
Comment 8 Brendan Molloy 2015-12-11 00:31:50 UTC
I tried the netwait method:

Dec 11 11:27:33 aerie kernel: Waiting for ue0 to have link
Dec 11 11:27:33 aerie kernel: /etc/rc: ERROR: ifconfig ue0 failed
Dec 11 11:27:33 aerie kernel: Mounting NFS file systems:[tcp] 10.0.0.4:/nfs/BorgBackups: RPCPROG_NFS: RPC: Port mapper failure - RPC: Unable to send

The interface does not exist at all until after the script had already run, so maybe the script should have an option to be more patient with errors?
Comment 9 Brendan Molloy 2015-12-11 14:09:28 UTC
Because I love FreeBSD, I am providing further information on this issue at 1am on a Saturday on Hans' and Kubilay's behalf.

I was asked to add hw.usb.axe.debug=16 to sysctl.conf, and I rebooted.

The only difference in logs was this:

Dec 12 01:01:34 aerie kernel: ue0: <USB Ethernet> on axe0
Dec 12 01:01:34 aerie kernel: ue0: Ethernet address: 00:50:b6:16:30:de
Dec 12 01:01:35 aerie ntpd[601]: ntpd 4.2.8p3-a (1): Starting
Dec 12 01:01:35 aerie kernel: .
Dec 12 01:01:36 aerie kernel: axe_bulk_write_callback: transfer complete
Dec 12 01:02:05 aerie last message repeated 10 times
Dec 12 01:02:53 aerie last message repeated 28 times
Comment 10 Ian Lepore freebsd_committer freebsd_triage 2015-12-11 16:22:22 UTC
(In reply to Brendan Molloy from comment #8)

Oh!  I should have realized... the interface doesn't exist until devd comes along and creates it, and that's too late.  I think you can fix that by adding to /boot/loader.conf:

  if_axe_load=YES

and you'll probably still need either SYNCDHCP or newait_enable for static ip.
Comment 11 Brendan Molloy 2015-12-12 05:27:25 UTC
(In reply to Ian Lepore from comment #10)

I tried this, and you are right, I still needed netwait.

The patch I provided in bug #205263 makes netwait more resilient to late loading of the modules, as it doesn't immediately scream about no interface when no iface is yet found. The timeout then gives enough time for the module to load at its own pace, while not requiring any changes to /boot/loader.conf.

If my patch is clean enough, it provides a solution inline with the principle of least astonishment to a problem that may become more prevalent in the future, as many modern laptops do not any longer include ethernet ports and users rely on USB or Thunderbolt ethernet adapters.
Comment 12 Ian Lepore freebsd_committer freebsd_triage 2015-12-17 23:50:45 UTC
Created attachment 164335 [details]
Updated diff for netwait script

I was updating the comments at the top of the script (seems to be the only "documentation" for netwait), and while describing how you could provide a list of interfaces to wait for, I realized it only allowed 1 interface.  I figured someone could have usb wifi and usb wired NIC or multiple NICs or whatever, so I updated the script to handle a list of interfaces.

I can only test the failure path (interface never arrives), it needs a test with real late-arriving hardware.
Comment 13 Ian Lepore freebsd_committer freebsd_triage 2015-12-18 01:04:11 UTC
I have turned the latest diff into a review at https://reviews.freebsd.org/D4608
Comment 14 commit-hook freebsd_committer freebsd_triage 2015-12-26 18:22:10 UTC
A commit references this bug:

Author: ian
Date: Sat Dec 26 18:21:33 UTC 2015
New revision: 292752
URL: https://svnweb.freebsd.org/changeset/base/292752

Log:
  Enhance rc.d/netwait script to wait for late-attaching interfaces such as
  USB NICs.

  USB network hardware may not be enumerated and available when the rc.d
  networking scripts run. Eventually the USB attachment completes and devd
  events cause the network initialization to happen, but by then other rc.d
  scripts have already failed, because services which depend on NETWORKING
  (such as mountcritremote) may end up running before the network is actually
  ready.

  There is an existing netwait script, but because it is dependent on
  NETWORKING it runs too late to prevent failure of some other rc
  scripts. This change flips the order so that NETWORKING depends on netwait,
  and netwait now depends on devd and routing (the former is needed to make
  interfaces appear, and the latter is needed to run the ping tests in
  netwait).

  The netwait script used to be oriented primarily towards "as soon as any
  host is reachable the network is fully functional", so you gave it a list of
  IPs to try and you could optionally name an interface and it would wait for
  carrier on that interface. That functionality still works the same, but now
  you can provide a list of interfaces to wait for and it waits until each one
  of them is available. The ping logic still completes as soon as the first IP
  on the list responds.

  These changes were submitted by Brenden Molloy <brendan+freebsd@bbqsrc.net>
  in PR 205186, and lightly modified by me to allow a list of interfaces
  instead of just one.

  PR:		205186
  Differential Revision:	https://reviews.freebsd.org/D4608 (timeout w/o review)

Changes:
  head/etc/defaults/rc.conf
  head/etc/rc.d/NETWORKING
  head/etc/rc.d/netwait
Comment 15 commit-hook freebsd_committer freebsd_triage 2016-01-24 19:41:36 UTC
A commit references this bug:

Author: ian
Date: Sun Jan 24 19:41:32 UTC 2016
New revision: 294680
URL: https://svnweb.freebsd.org/changeset/base/294680

Log:
  MFC r292752:

    Enhance rc.d/netwait script to wait for late-attaching interfaces such as
    USB NICs.

    USB network hardware may not be enumerated and available when the rc.d
    networking scripts run. Eventually the USB attachment completes and devd
    events cause the network initialization to happen, but by then other rc.d
    scripts have already failed, because services which depend on NETWORKING
    (such as mountcritremote) may end up running before the network is actually
    ready.

    There is an existing netwait script, but because it is dependent on
    NETWORKING it runs too late to prevent failure of some other rc
    scripts. This change flips the order so that NETWORKING depends on netwait,
    and netwait now depends on devd and routing (the former is needed to make
    interfaces appear, and the latter is needed to run the ping tests in
    netwait).

    The netwait script used to be oriented primarily towards "as soon as any
    host is reachable the network is fully functional", so you gave it a list of
    IPs to try and you could optionally name an interface and it would wait for
    carrier on that interface. That functionality still works the same, but now
    you can provide a list of interfaces to wait for and it waits until each one
    of them is available. The ping logic still completes as soon as the first IP
    on the list responds.

    These changes were submitted by Brenden Molloy <brendan+freebsd@bbqsrc.net>
    in PR 205186, and lightly modified by me to allow a list of interfaces
    instead of just one.

  PR:           205186
  Relnotes:	yes

Changes:
_U  stable/10/
  stable/10/etc/defaults/rc.conf
  stable/10/etc/rc.d/NETWORKING
  stable/10/etc/rc.d/netwait
Comment 16 neverendservices 2023-05-29 11:12:36 UTC
MARKED AS SPAM
Comment 17 Rockypatel 2023-07-09 14:54:20 UTC
MARKED AS SPAM
Comment 18 VN Editor For PC 2023-07-17 09:51:14 UTC
MARKED AS SPAM
Comment 19 ctbae 2023-08-08 06:16:32 UTC
MARKED AS SPAM
Comment 20 Dicik 2023-08-08 09:16:41 UTC
MARKED AS SPAM
Comment 21 streetgirls69 2023-08-08 09:25:18 UTC
MARKED AS SPAM
Comment 22 ctmot 2023-08-08 09:32:37 UTC
MARKED AS SPAM
Comment 23 depadas 2023-08-08 09:38:03 UTC
MARKED AS SPAM
Comment 24 Free Story 2023-09-06 07:10:52 UTC
MARKED AS SPAM
Comment 25 My Heaven Models Bangalore 2023-09-06 07:14:53 UTC
MARKED AS SPAM
Comment 26 My Heaven Models Bangalore 2023-09-06 07:16:33 UTC
MARKED AS SPAM
Comment 27 indiamyheavenmodels 2023-09-08 06:19:17 UTC
MARKED AS SPAM
Comment 28 Hyderabad Lily 2023-09-08 07:22:53 UTC
MARKED AS SPAM
Comment 29 priyanka 2023-09-08 08:14:55 UTC
MARKED AS SPAM
Comment 30 My Heaven Models Bangalore 2023-09-11 10:16:45 UTC
MARKED AS SPAM
Comment 31 visakhapatnam escorts 2023-09-11 13:25:51 UTC
MARKED AS SPAM
Comment 32 mussoorie myheavenmodels 2023-09-14 11:29:43 UTC
MARKED AS SPAM
Comment 33 priyanka 2023-09-16 07:35:34 UTC
MARKED AS SPAM
Comment 34 eatandrunpolice123 2023-09-20 10:09:00 UTC
MARKED AS SPAM
Comment 35 Kavya Gupta 2023-09-30 10:01:50 UTC
MARKED AS SPAM
Comment 36 Russian Escorts In Mumbai 2023-10-06 09:33:07 UTC
MARKED AS SPAM
Comment 37 Ethan carter 2023-10-11 05:49:04 UTC
MARKED AS SPAM
Comment 38 A Jaipur Escort 2023-10-12 12:41:20 UTC
MARKED AS SPAM
Comment 39 tinaduuta 2023-10-13 10:03:49 UTC
MARKED AS SPAM
Comment 40 tinaduuta 2023-10-13 10:04:34 UTC
MARKED AS SPAM
Comment 41 renudas 2023-10-17 09:48:00 UTC
MARKED AS SPAM
Comment 42 anna 2023-10-31 13:58:28 UTC
MARKED AS SPAM
Comment 43 Chhavi Garg 2023-12-07 05:41:17 UTC
MARKED AS SPAM
Comment 44 garimasuri 2023-12-08 05:45:06 UTC
MARKED AS SPAM
Comment 45 xedifo 2023-12-21 14:05:46 UTC
MARKED AS SPAM
Comment 46 Aashvi Khanna 2024-01-02 10:14:35 UTC
MARKED AS SPAM
Comment 47 isabasu 2024-01-08 10:03:32 UTC
MARKED AS SPAM
Comment 48 Neha Mari 2024-01-11 09:55:09 UTC
MARKED AS SPAM
Comment 49 Divya Aggarwal 2024-01-23 18:49:10 UTC
MARKED AS SPAM
Comment 50 Haridwar Escorts Dimpal 2024-01-29 15:56:58 UTC
MARKED AS SPAM
Comment 51 Haridwar Escorts Dimpal 2024-02-19 09:07:26 UTC
MARKED AS SPAM
Comment 52 India Escorts 2024-02-19 17:31:11 UTC
MARKED AS SPAM
Comment 53 India Escorts 2024-02-19 18:42:10 UTC
MARKED AS SPAM
Comment 54 India Escorts 2024-02-19 18:57:14 UTC
MARKED AS SPAM
Comment 55 India Escorts 2024-02-19 19:13:24 UTC
MARKED AS SPAM
Comment 56 agencymumbaigirl 2024-02-28 09:37:18 UTC
MARKED AS SPAM
Comment 57 riyamalik2u 2024-03-11 08:17:54 UTC
MARKED AS SPAM
Comment 58 Tina Kapoor 2024-03-12 06:37:59 UTC
MARKED AS SPAM
Comment 59 shruti goyal 2024-03-15 07:07:35 UTC
MARKED AS SPAM
Comment 60 shruti goyal 2024-03-15 07:08:10 UTC
MARKED AS SPAM
Comment 61 shruti goyal 2024-03-15 07:08:31 UTC
MARKED AS SPAM
Comment 62 shruti goyal 2024-03-15 07:08:49 UTC
MARKED AS SPAM
Comment 63 shruti goyal 2024-03-15 07:09:21 UTC
MARKED AS SPAM
Comment 64 shruti goyal 2024-03-15 07:10:26 UTC
MARKED AS SPAM
Comment 65 riyamalik2u 2024-03-15 07:10:53 UTC
MARKED AS SPAM
Comment 66 shruti goyal 2024-03-15 07:12:05 UTC
MARKED AS SPAM
Comment 67 vipmumbaiescorts 2024-04-06 04:58:34 UTC
MARKED AS SPAM
Comment 68 call girl mumbai 2024-04-07 08:50:44 UTC
MARKED AS SPAM
Comment 69 Mumbai Escorts 2024-04-10 15:06:26 UTC
MARKED AS SPAM
Comment 70 Tina Ji 2024-04-11 07:48:57 UTC
MARKED AS SPAM
Comment 71 Tina Ji 2024-04-11 07:49:54 UTC
MARKED AS SPAM
Comment 72 Monaroy 2024-04-17 18:18:44 UTC
MARKED AS SPAM
Comment 73 anna 2024-05-15 12:31:08 UTC
MARKED AS SPAM
Comment 74 punebeauties1 2024-06-03 04:55:44 UTC
Your blog provided me with the information I was looking for. Having this kind of happiness is unimaginable. Thanks for posting this. I really appreciate it
https://www.punebeauties.com
https://www.punebeauties.com/call-girls-pune.html
Comment 75 Girls Club 2024-06-05 13:48:46 UTC
https://loveleenmalhotra.com/
https://bestcallgirls.in/
https://www.ayeshakaur.com
https://www.kiranbajaj.com/
https://girlsclub.in/
https://in.delhigirlsclub.com/call-girls/delhi/
https://jazlyn.in/
https://callgirlinchandigarh.com/
https://muskankhan.com/
https://www.ziyaa.net/
https://www.chandigarhcallgirlsagency.co.in/
https://www.naziaa.com/
https://callgirlsinzirakpur.in/
https://www.anchalkapoor.com/
https://www.honeytyagi.com/
https://www.zoya.club/
https://www.mohalicallgirls.com/
https://callgirls4you.com/
https://www.mohalicallgirlsagency.in/
https://www.sarakaur.in/
https://www.honeytyagi.com/bangalore-escorts/
https://muskankhan.com/
https://www.anusaxena.com
https://www.sonaidas.com
https://www.deepalimehra.com/
https://www.navleenkaur.com/
https://muskankhan.com/delhi-call-girls/
https://muskankhan.com/noida-call-girls/
https://muskankhan.com/greater-noida-call-girls/
https://in.delhigirlsclub.com/call-girls/aerocity/
https://in.delhigirlsclub.com/call-girls/dehradun/
https://in.delhigirlsclub.com/call-girls/goa/
https://in.delhigirlsclub.com/call-girls/gurgaon/
https://in.delhigirlsclub.com/call-girls/ludhiana/
https://in.delhigirlsclub.com/call-girls/mohali/
https://in.delhigirlsclub.com/call-girls/noida/
https://www.preetkaur.com/
https://girlsclub.in/call-girl-noida.html
https://girlsclub.in/bangalore-call-girls.html
https://girlsclub.in/call-girls-chandigarh.html
https://girlsclub.in/call-girl-gurgaon.html
https://girlsclub.in/dehati-sexy-video-list.html
https://girlsclub.in/red-light-areas-in-hyderabad.html
https://girlsclub.in/red-light-areas-in-delhi.html
https://girlsclub.in/call-girl-greater-noida.html
Comment 76 Heera 2024-06-05 14:32:38 UTC
If u need Most Beautiful Call Girls in Islamabad so Than Call me. I am Provide u Brilliant Option. So Don't Waste your Time and Call me. https://callgirlsinislamabad.agency/
Comment 77 Payal Kapoor 2024-06-08 06:36:54 UTC
Yeah surely, just like other people, these escorts in Chandigarh get offended by rude behaviour of clients. If you don’t want to offend them and enjoy the best services of your life, then you should treat them with respect, and you should make sure that you are providing them happiness too. 
https://chdescorts.com
https://bit.ly/m/chdgirl
https://gravatar.com/payalkapoor997
https://www.producthunt.com/@hotpayalkapoor
Comment 78 Nisha Bhat 2024-06-08 12:01:19 UTC
Nisha Bhat is a pure Independent Chandigarh Escorts girl in Chandigarh Call Girls City. She is beautiful hot and sizzling girl.

Hi, I am Nisha Bhat Model Girl From Chandigarh at Nisha Bhat Agency. 

Call Girl in Chandigarh Nisha Bhat Provide INCall facility ₹,2500 with free door step delivery. 100% satisfaction book for day or whole night 24*7.

The most classy, top notch and high profile Chandigarh Escorts Service at Affordable Rates with best female Models and High Profile Girls.

https://nishabhat.in
https://nishabhat.in/gallery.html
https://nishabhatescort.wordpress.com/
https://nishabhatescort.wordpress.com/2024/06/08/feel-the-care-of-the-escorts-in-chandigarh/
https://nishabhatescort.mystrikingly.com/blog/chandigarh-escorts-provide-the-extreme-thrill
https://zirakpurescortservice.site123.me/forum/enjoy-the-best-sexual-time-with-the-call-girls-in-zirakpur
https://www.palscity.com/nishabhathot
https://www.ptats.co.id/Community/nishabhathot
https://callgirlschandigarhagency.weebly.com/
https://nishabhatescort.mystrikingly.com/blog/chandigarh-escort-agency-provides-satisfaction
https://nishabhatescort.mystrikingly.com/
https://securityheaders.com/?q=https%3A%2F%2Fnishabhat.in&followRedirects=on
https://www.peepso.com/external-link/?url=https%3A%2F%2Fnishabhat.in
Comment 83 Kolkata Dream Escorts 2024-06-18 09:05:08 UTC
Hello, I am Kolkata Dream Girl, a-list free Model in Kolkata  giving outcall and Incall organizations, private arousing meeting and back rub organizations for obliging colleagues. In case you are a decent man looking for something reasonably interesting and uncommon for your outing with Call girls in Kolkata, by then, here I am.
http://www.kolkatadreamescorts.com
http://www.kolkatasexyescorts.com
http://www.payalkolkataescort.in
http://www.elitekolkataescorts.co.in
http://www.kolkatatopescortservices.com
Comment 84 Mumbai Escorts Sana 2024-06-19 23:49:13 UTC
My name is Sana Kaur and I am a super cute escort in Mumbai with brown hair and great personality. Sana Kaur is a sweet personality for your night. I am a very attractive girl with a cheerful disposition, who is eager to love you in any way. When you meet me, we will exchange love and spend quality time together. Come to Mumbai to experience the best escort service in Mumbai. I will please you and give you all my warmth. If you meet me, you’ll forget all the rest. Come meet me soon and experience all the love and happiness in one night. I am ready to meet the needs of my clients and guarantee you will be happy. https://www.sanakaur.com/
Comment 86 swati sharma 2024-06-20 08:32:10 UTC
It is because of these ingredients that most people have visited this region of the world. Ahmedabad Escort Girls is the most popular incredible service and is what people are looking for. It is about getting the same experience but also different things. Today, one must select the best ingredients for incredible services that have become extremely popular and it's all about enjoying yourself as well as having a fulfilling experience.
https://www.adita.in
https://www.adita.in/vadodara-escorts.html
https://www.adita.in/ahmedabad-escorts.html
https://www.adita.in/shimla-escorts.html
Comment 87 Poonampanday 2024-06-20 12:59:32 UTC
We offer high-profile model Jaipur call girls and escort services in Jaipur. In Jaipur, there are numerous escort service providers. The majority are deceptions. Because they demanded payment in advance. Our call girl in Jaipur did not ask for any kind of advance fees from the client Escort Service Jaipur. “Jaipur Escorts” is a VIP Jaipur Call girls, the level and reputation of which have long established themselves as the most reliable and high quality. Over the years of working with call girls in Jaipur, our Jaipur escort agency has built a base of charming, smart, and sociable girls for elite escorts, who know the difficult life of business people.

https://lovemenights.com/
https://amyescorts.com/
https://sweetamalik.com/
Comment 91 Dolly Saxena 2024-06-24 07:30:33 UTC
Dolly saxena is a model girls. This post makes me very happy, and all the details are fantastic.
https://dollysaxena.com/
https://dollysaxena.com/shastri-nagar-escorts.html
https://dollysaxena.com/inder-lok-escorts.html
https://dollysaxena.com/kanhaiya-nagar-escorts.html
https://dollysaxena.com/keshav-puram-escorts.html
https://dollysaxena.com/netaji-subhash-place-escorts.html
https://dollysaxena.com/kohat-enclave-escorts.html
https://dollysaxena.com/pitam-pura-escorts.html
https://dollysaxena.com/rohini-east-escorts.html
https://dollysaxena.com/rohini-west-escorts.html
https://dollysaxena.com/rithala-escorts.html
https://dollysaxena.com/shaheed-sthal-escorts.html
https://dollysaxena.com/hindon-escorts.html
https://dollysaxena.com/arthala-escorts.html
https://dollysaxena.com/mohan-nagar-escorts.html
https://dollysaxena.com/shyam-park-escorts.html
https://dollysaxena.com/major-mohit-sharma-escorts.html
https://dollysaxena.com/raj-bagh-escorts.html
https://dollysaxena.com/shaheed-nagar-escorts.html
https://dollysaxena.com/samaypur-badli-escorts.html
https://dollysaxena.com/rohini-sector-18-19-escorts.html
https://dollysaxena.com/haiderpur-badli-mor-escorts.html
https://dollysaxena.com/jahangir-puri-escorts.html
https://dollysaxena.com/adarsh-nagar-escorts.html
https://dollysaxena.com/azadpur-escorts.html
https://dollysaxena.com/gtb-nagar-escorts.html
https://dollysaxena.com/rajiv-chowk-escorts.html
https://dollysaxena.com/vishwavidyalaya-escorts.html
Comment 96 bhopalgirlhub 2024-07-01 12:02:18 UTC
If you want to use Exotic Call Girl Service in Bhopal, you can take me to Outcall Service at your hotel. Bhopal Russian Call Girls, Get the most perfect experience with a foreign call girl. https://bhopalgirlhub.in/