Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-28
19:30:48 efried mriedem, dansmith: Didn't y'all see this the other day ^ ?
19:31:01 mriedem no it was an instance.save() when i saw it
19:32:18 smcginnis Odd that it seems to be somewhere that it kills the whole service.
19:32:25 mriedem in this case, compute is asking the cell conductor for the min nova-compute service version in that cell,
19:32:30 mriedem and timing out waiting for a reply
19:32:40 mriedem smcginnis: it's on startup of the service - init the compute rpc api
19:32:59 mriedem i wonder if nova-compute is starting before nova-cond_cell1?
19:33:09 mriedem n-cond-cell1 i mean
19:33:22 smcginnis Was just checking timestamps there.
19:33:23 openstackgerrit Eric Fried proposed openstack/nova master: Do test_reshape with an actual startup https://review.openstack.org/597218
19:33:47 mriedem n-cond-cell1 appears to be up around at least Aug 27 14:30:40.185973
19:34:03 mriedem n-cpu fails at Aug 27 14:31:35.678991
19:34:16 smcginnis Yeah, about a second ahead of the failure on another one I'm looking at too.
19:34:34 mriedem so n-cpu and n-cond-cell1 looks like they are starting at roughly the same time,
19:34:43 mriedem n-cpu starts querying n-cond-cell1 before it's fully up maybe
19:34:46 mriedem then times out
19:35:27 smcginnis Does that need to happen in __init__? Seems like maybe somewhere after where it could retry once or twice might be good.
19:36:24 mriedem doesn't probably have to no
19:36:29 mriedem we could do it in post_start_hook or something
19:36:32 mriedem not sure if it would help
19:37:15 mriedem self.compute_rpcapi is pretty much only used for inter-compute comms anyway
19:37:20 mriedem it's not used on startup
19:38:32 mriedem 2018-08-27 14:30:31.159 | + functions-common:_run_under_systemd:1482 : sudo systemctl start devstack@n-cond-cell1.service
19:38:52 mriedem 2018-08-27 14:30:32.414 | + functions-common:_run_under_systemd:1482 : sudo systemctl start devstack@n-cpu.service
19:39:19 mriedem i think using upgrade_levels=auto is relatively new in devstack, and i think i added it...
19:39:36 smcginnis I'm fine with blaming you.
19:40:01 mriedem https://github.com/openstack-dev/devstack/commit/21221d1ad1462cdcaed4d052c3324ae384b407d4
19:40:17 mriedem been there since may so not new
19:40:32 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22_determine_version_cap%5C%22%20AND%20message%3A%5C%22MessagingTimeout%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d
19:40:40 mriedem but we might not have multi-line indexing properly in this job
19:41:11 mriedem here we go http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22_determine_version_cap%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d
19:41:26 mriedem smcginnis: can you report a bug for this so we can at least start tracking it in e-r?
19:41:40 melwitt internally, people have asked for retry behavior there, if nova-compute starts up before nova-conductor is available
19:41:54 melwitt I looked at it a bit but didn't see where/how we could add it
19:42:14 smcginnis OK, I'll file a bug so at least it's tracked.
19:42:31 mriedem i'd probably move it to post_start_hook,
19:42:35 mriedem add a 3-time retry on it,
19:42:41 mriedem fail if we still can't get the thing
19:42:52 mriedem threading it out doesn't help since then we can't kill the service
19:43:17 mriedem although i think you can just SIGHUP and get nova-compute back on track after n-cond-cell1 is runnign
19:43:31 mriedem i know a guy in portland that knows how all of this shit works
19:43:48 melwitt :)
19:48:24 smcginnis Lovely - launchpad timeout submitting bug.
19:49:05 openstackgerrit Eric Fried proposed openstack/nova master: reshaper gabbit: Nix comments re doubled max_unit https://review.openstack.org/597220
19:49:53 openstack Launchpad bug 1789484 in OpenStack Compute (nova) "n-cpu fails init on timeout calling n-cond-cell1" [Undecided,New]
19:49:53 smcginnis Done - https://bugs.launchpad.net/nova/+bug/1789484
19:55:53 smcginnis mriedem: This looks like it started on the 18th - http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22_determine_version_cap%5C%22%20AND%20tags:%5C%22screen-n-cpu.txt%5C%22&from=7d
20:06:14 mriedem efried: comments and nits inline https://review.openstack.org/#/c/576236/
20:06:24 efried mriedem: ack, thx
20:06:26 mriedem smcginnis: we only save 10 days of logs
20:06:42 smcginnis Well poop
20:55:46 openstackgerrit Eric Fried proposed openstack/nova master: Fail heal_allocations if placement is borked https://review.openstack.org/597237
20:55:55 efried mriedem: ^
20:59:00 mriedem mmm, bork
21:04:51 mriedem efried: i need a retry decorator, do you remember any pros of retrying.retry over oslo's RetryDecorator?
21:05:08 efried mriedem: yeah, lemme context switch...
21:05:51 efried mriedem: I think the main advantage of retrying is: no threading.
21:06:14 efried But RetryDecorator has a simpler/better-documented interface
21:10:13 mriedem ok that's about what i remembered
21:10:21 mriedem during the RT._update thing
21:15:54 melwitt mriedem: I've been using a script for counting blueprint stuff for future charting, if you wouldn't mind reviewing https://review.openstack.org/592628 (second patch in series)
21:28:06 mriedem +2
21:28:59 efried mriedem: Responded to your questions on https://review.openstack.org/#/c/576236/
21:29:22 efried mriedem: Let me know if you want anything acted on immediately, or if fups are okay.
21:29:35 mriedem mmm f ups
21:29:39 mriedem yeah was just opening that
21:29:57 mriedem there is also a strange small human in my house now since school is out so i'll be potentially distracted
21:30:58 efried mriedem: No huge hurry, I think I'm probably going to need to knock off for the evening soon.
21:31:24 melwitt mriedem: thanks. the counting blueprint script is the second patch stacked on top of that one, though. pls
21:31:52 mriedem oh right...
21:40:27 mriedem efried: +2 on that
21:40:31 mriedem https://review.openstack.org/#/c/576236/ that is
21:40:41 mriedem jaypipes was +2 so i suppose i could +W
21:40:41 efried mriedem: Cool, thx
21:40:47 efried yahsure
21:40:50 mriedem jaypipes: do you want to +W https://review.openstack.org/#/c/576236/ ?
21:40:56 mriedem or happy to have me proxy
21:40:59 efried that would put the whole pile in the gate.
21:41:11 jaypipes mriedem: done
21:42:02 efried woot
21:45:10 openstackgerrit Eric Fried proposed openstack/nova master: Name arguments to _get_provider_ids_matching https://review.openstack.org/597291
22:24:20 openstackgerrit Eric Fried proposed openstack/nova master: Other host allocs may appear in gafpt during evac https://review.openstack.org/597301
22:37:54 openstackgerrit Eric Fried proposed openstack/nova master: Mention (unused) RP generation in POST /allocs/{c} https://review.openstack.org/597304
22:54:07 openstackgerrit melanie witt proposed openstack/nova-specs master: Propose configurable maximum number of volumes to attach https://review.openstack.org/597306
23:34:32 openstackgerrit Merged openstack/nova master: Don't use '_TransactionContextManager._async' https://review.openstack.org/597173
23:34:38 openstackgerrit Merged openstack/nova master: Revert "Don't use '_TransactionContextManager._async'" https://review.openstack.org/597174
23:40:32 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (4) https://review.openstack.org/574106
23:40:45 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (5) https://review.openstack.org/574110
23:40:57 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (6) https://review.openstack.org/574113
23:41:09 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (7) https://review.openstack.org/574974
23:41:24 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (8) https://review.openstack.org/575311
23:41:45 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (9) https://review.openstack.org/575581
23:43:51 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (10) https://review.openstack.org/576017
23:43:59 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (11) https://review.openstack.org/576018
23:44:23 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (12) https://review.openstack.org/576019
23:44:52 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13) https://review.openstack.org/576020
23:45:15 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.openstack.org/576027
23:45:32 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.openstack.org/576031
23:45:54 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.openstack.org/576299
23:46:16 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.openstack.org/576344

Earlier   Later