Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-25
15:41:25 mriedem no, you just need to stub the fake notifier stuff
15:41:46 dansmith okay, I got that of course and am getting the notifs but payloads are empty
15:41:50 dansmith maybe I'm emitting wrong
15:46:22 dansmith must be because the other ones have payloads
15:46:23 dansmith danigt
15:48:07 dansmith helps if I register the payload too :D
15:48:47 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Fail compute service delete if resource provider delete fails https://review.opendev.org/678100
15:49:58 mriedem ^ is wip until i get some reviews on what's proposed there which is based on what was discussed in the ML for not orphaning resource providers when deleting a compute service that still has allocations against it
15:50:15 mriedem we continue to get bugs in triage for this kind of thign
15:50:28 mriedem e.g. https://bugs.launchpad.net/nova/+bug/1849701
15:50:29 openstack Launchpad bug 1849701 in OpenStack Compute (nova) "Resource_provider entry related to a deleted compute node, unable to migrate vms to the node" [Undecided,New]
16:06:45 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.opendev.org/602201
16:09:33 mriedem looks like docs job is busted
16:10:23 cdent dansmith, mriedem : when nova-compute comes up and tries to initiate talking to the conductor it uses an "early_timeout" and then after 10 attempts fails over to the global timeout. If those 10 attempts pass, is it possible the nova-compute will sit doing nothing on that long timeout for a long time before talking to the conductor, or should it loop quickly if there is an actual connection failed (rather than timing out
16:10:23 cdent waiting for response)? I'm reading some bugs where it seems like the sit doing nothing is happening, but it seems illogical.
16:10:24 mriedem https://zuul.opendev.org/t/openstack/build/10344924bca44849b06b8fae29b5494f/log/job-output.txt#652
16:11:22 mriedem cdent: not sure off the top of my head, probably a better question for dansmith
16:13:25 cdent in placement we addressed that docs issue by having a different set of requirements for docs
16:13:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Enable evacuation with qos ports https://review.opendev.org/688688
16:14:00 openstackgerrit Matt Riedemann proposed openstack/nova master: Allow evacuating server with port resource request https://review.opendev.org/688387
16:14:08 dansmith cdent: not sure what you mean by early timeout
16:14:18 cdent dansmith: one sec, will get code link
16:14:20 dansmith cdent: it should be calling ping for the conductor pretty regularly and wait forever
16:14:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Enable evacuation with qos ports https://review.opendev.org/688688
16:15:30 cdent dansmith: https://github.com/openstack/nova/blob/master/nova/conductor/api.py#L41
16:16:05 dansmith oh right
16:16:11 cdent after early attempts expires, if the global rpc timeout is huge (because people do that...) it appears like it can be a very long time before "ready" happens
16:16:43 cdent but I would have thought that if the conductor is literally not there, you would know sooner
16:16:50 dansmith cdent: people who set the global timeout large are begging for problems, so I don't think we should do something different here
16:16:51 dansmith however,
16:17:25 mriedem vmware sets the timeout large because they are pulling 1K+ instances from conductor on startup yeah?
16:17:26 cdent but could it be that if the message bus is up but the conductor is not, the ping will hang out in rabbit... waiting
16:17:28 dansmith the reason for that was because if you have a power outage and start a thousand computes, they need to chill if they can't get a response right away.. just because they have a connection doesn't mean they're not overwhelming the conductor(s), which is why we have the backoff
16:17:44 dansmith cdent: sure
16:17:47 cdent mriedem: probably something like that, yeah
16:18:13 cdent Okay, cool, this confirms what I thought was happening, but hadn't been able to trace it all the way down
16:18:36 cdent mriedem: there also appears to be an ordering problem in service restarts
16:19:01 cdent thanks to you both
16:19:10 cdent I'll go write down more stuff that people will ignore...
16:19:17 cdent (not you, I mean internal people)
16:19:32 mriedem *those* people
16:19:35 openstackgerrit John Garbutt proposed openstack/nova master: WIP: Integrating with unified limits https://review.opendev.org/615180
16:19:51 cdent them
16:24:10 mriedem blarg, yeah we have doc/requirements.txt but https://review.opendev.org/#/c/689152/ threw requirements.txt back into the install for the docs job
16:24:11 mriedem fixing
16:26:36 openstackgerrit Dan Smith proposed openstack/nova master: Log some stats for image pre-cache https://review.opendev.org/688173
16:26:37 openstackgerrit Dan Smith proposed openstack/nova master: Add Aggregate image caching progress notifications https://review.opendev.org/691390
16:29:03 mriedem https://bugs.launchpad.net/nova/+bug/1849870
16:29:03 openstack Launchpad bug 1849870 in OpenStack Compute (nova) "docs build failing on psycopg2 dependency" [Undecided,New]
16:39:42 openstackgerrit Matt Riedemann proposed openstack/nova master: Stop building docs with (test-)requirements.txt https://review.opendev.org/691391
16:39:43 mriedem ^ will hopefully unblock the gate for nova ^
16:41:48 dansmith mriedem: should we slam that in?
16:42:11 mriedem let's see if it passes the docs job first, it works for me locally
16:42:17 mriedem guess it could take awhile to get a node...
16:42:22 dansmith okay
16:42:35 mriedem oh it got a node http://zuul.openstack.org/stream/63d6bbd171134e00877ae6ce61856a84?logfile=console.log
16:42:42 mriedem build baby build!
16:47:39 mriedem dansmith: looks like it passed so you can commence to slamming
16:48:04 dansmith mriedem: you going to fix the typo?
16:48:21 mriedem sure, sec
16:48:22 dansmith or just clarifying for review?
16:48:25 dansmith ack
16:49:03 openstackgerrit Matt Riedemann proposed openstack/nova master: Stop building docs with (test-)requirements.txt https://review.opendev.org/691391
16:50:32 openstackgerrit Dan Smith proposed openstack/nova master: Add Aggregate image caching progress notifications https://review.opendev.org/691390
17:00:44 mriedem melwitt: were you going to backport https://review.opendev.org/#/c/688206/ and the one below it?
17:00:55 mriedem i'm planning a train release next week so probably good to get that in it
17:01:14 melwitt mriedem: yes. I wasn't sure if I should wait for the top patch to merge first before proposing
17:01:28 mriedem i don't think you need to wait
17:01:33 melwitt ok
17:09:25 dansmith mriedem: what's the trick for generating the notification samples?
17:32:48 mriedem you have to create them manually
17:33:02 mriedem the json file i mean
17:33:18 dansmith WAT
17:33:50 mriedem all sales are final
17:35:00 dansmith if I add this is there something that will check my work?
17:35:26 mriedem the functional test should be comparing against the sample
17:35:39 dansmith oh if I write that test okay
17:36:04 dansmith I should rebase mine on yours I guess
17:36:05 mriedem yeah so instead of this https://review.opendev.org/#/c/691390/2/nova/tests/functional/compute/test_cache_image.py
17:36:08 mriedem right
17:36:17 dansmith not instead, in addition right?
17:36:26 mriedem i don't see why we'd need both
17:37:01 dansmith because it already has the error case and other-agg case
17:37:04 mriedem this is where the test in mine validates the sample against the actual notification generated by the test https://review.opendev.org/#/c/691129/1/nova/tests/functional/notification_sample_tests/test_aggregate.py@199
17:37:28 mriedem oh yeah ok that makes sense
17:39:22 openstackgerrit Merged openstack/nova stable/train: libvirt: Ignore volume exceptions during post_live_migration https://review.opendev.org/691281
17:40:25 openstackgerrit melanie witt proposed openstack/nova stable/train: Add regression test for bug 1824435 https://review.opendev.org/691402
17:40:25 openstack bug 1824435 in OpenStack Compute (nova) "fill_virtual_interface_list migration fails on second attempt" [Medium,In progress] https://launchpad.net/bugs/1824435 - Assigned to melanie witt (melwitt)
17:40:25 openstackgerrit melanie witt proposed openstack/nova stable/train: Remove redundant call to get/create default security group https://review.opendev.org/691403
17:43:35 openstackgerrit Merged openstack/nova stable/queens: Fix unit of hw_rng:rate_period https://review.opendev.org/689155
17:50:56 dansmith mriedem: I don't really understand why we specify the payload in a file, but also some fields of the payload to verify in the doc schema file or whatever
17:51:25 dansmith mriedem: this stuff: https://review.opendev.org/#/c/691129/1/doc/notification_samples/aggregate-cache_images-end.json@5
18:04:56 openstackgerrit Dan Smith proposed openstack/nova master: Add notification sample test for aggregate.cache_images.start|end https://review.opendev.org/691129
18:04:56 openstackgerrit Dan Smith proposed openstack/nova master: Log some stats for image pre-cache https://review.opendev.org/688173
18:04:57 openstackgerrit Dan Smith proposed openstack/nova master: Add Aggregate image caching progress notifications https://review.opendev.org/691390
18:21:45 openstackgerrit Merged openstack/nova master: Stop building docs with (test-)requirements.txt https://review.opendev.org/691391
18:21:55 mriedem dansmith: that's an override for what's in the common file
18:22:06 mriedem otherwise you just get what's in "$ref": "common_payloads/AggregatePayload.json#",
18:22:30 dansmith so I can just put my stuff in the common file? I mean, that seemed to work fine, but didn't know if it wasn't "right"
18:23:11 dansmith if so, the rev I just pushed may be acceptable then

Earlier   Later