| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-06 | |||
| 17:01:46 | sean-k-mooney | in test_evacuate_with_soft_anti_affinity | |
| 17:02:14 | mriedem | i don't think that matters, | |
| 17:02:31 | mriedem | we're just calling GET /servers/server_id to get the status probably | |
| 17:02:32 | mriedem | WARNING [oslo.service.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 59.77 sec | |
| 17:02:47 | sean-k-mooney | likely not but just was wondering which one | |
| 17:02:56 | mriedem | chances are this is another infra node that hit some crazy io issue | |
| 17:03:28 | openstack | Launchpad bug 1851497 in OpenStack Compute (nova) "rng is only enabled if qemu guest agent is enabled" [Medium,Triaged] - Assigned to sean mooney (sean-k-mooney) | |
| 17:03:28 | sean-k-mooney | mriedem: oh that reminds me https://bugs.launchpad.net/nova/+bug/1851497 | |
| 17:03:57 | sean-k-mooney | the random number generator code has apparently been partly broken since queens ... | |
| 17:04:27 | sean-k-mooney | mriedem: did ye nail down the gate issue to io | |
| 17:04:46 | mriedem | sean-k-mooney: https://review.opendev.org/#/c/129325/ was kilo | |
| 17:04:50 | sean-k-mooney | if the vm runs out of entrapy it will have issue connecting to ssl sockets | |
| 17:04:53 | mriedem | oh wait | |
| 17:04:55 | mriedem | wrong change | |
| 17:05:09 | mriedem | oh no that's right | |
| 17:05:11 | mriedem | yeah kilo | |
| 17:05:31 | mriedem | who would have guessed that a 373 LOC refactor would break something.... | |
| 17:06:03 | sean-k-mooney | well no one noticed till now so ya | |
| 17:06:53 | sean-k-mooney | i was review the sev virtio viommu fix and found it odd that the qemu guest agent was adding a rng | |
| 17:07:35 | melwitt | mriedem: that reminds me, I was chatting with penick about the gate io issues and he suggested infra/tempest using a flavor where the iops qos/limit has been set. are you familiar with that? (I'm not) | |
| 17:10:09 | mriedem | the trend here is kind of interesting http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Function%20'nova.servicegroup.drivers.db.DbDriver._report_state'%20run%20outlasted%20interval%20by%5C%22&from=7d | |
| 17:10:14 | mriedem | noon and midnight | |
| 17:10:17 | mriedem | mostly on ovh | |
| 17:10:35 | efried | noisy cron job? | |
| 17:10:41 | mriedem | melwitt: i think we were talking about that in here the other day a bit | |
| 17:14:04 | amodi | mriedem++ melwitt++ thanks both!! | |
| 17:14:23 | mriedem | amodi: happy backporting | |
| 17:14:48 | amodi | lol sure :) | |
| 17:15:08 | mriedem | i posted to the ML about the ovh + spike cron | |
| 17:15:14 | mriedem | i was wondering about the dip there, but that's the weekend | |
| 17:16:13 | mriedem | so that query is kind of annoying because there are a lot that are just a couple of seconds, but those 50+ ones are indicative of something really bad on the node | |
| 17:16:43 | mriedem | mehinks it would be good to maybe have some config in oslo that sets a threshold for that and if it goes above the threshold then it's a WARNING but below it's just INFO? | |
| 17:16:52 | mriedem | right now everything is WARNING which is kind of just noise | |
| 17:16:59 | mriedem | bnemec: ^ what do you think about that? | |
| 17:17:26 | mriedem | like i'd like to make that a warning if something outlasts 30 seconds in devstack/gate and then we can trace it in e-r | |
| 17:18:27 | dansmith | mriedem: hmm, do we still need null_instance_uuid_scan? wasn't that like kilo and we have a constraint now? | |
| 17:19:25 | mriedem | heh yeah i think we can deprecate and remove that https://review.opendev.org/#/c/97946/ | |
| 17:19:32 | mriedem | https://review.opendev.org/#/q/topic:bp/db2-database+(status:open+OR+status:merged) !!! | |
| 17:19:46 | mriedem | classic riderman | |
| 17:37:55 | openstack | bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] https://launchpad.net/bugs/1788014 - Assigned to Matt Riedemann (mriedem) | |
| 17:37:55 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: DNM: break live migration to trigger bug 1788014 https://review.opendev.org/693045 | |
| 17:38:40 | jgwentworth | is there any way to discover whether force_config_drive=True in nova config from tempest? asking because of this tempest change https://review.opendev.org/693096 | |
| 17:43:23 | openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add nova-audit spec https://review.opendev.org/693226 | |
| 17:43:34 | artom | jgwentworth, nothing's discoverable in Tempest, it's all configured | |
| 17:43:46 | dansmith | mriedem: jgwentworth ^ wanna skim that for obvious booboos before I blast to the ML? | |
| 17:44:31 | jgwentworth | artom: yeah, I was thinking whether there's some kinda api discoverability from nova that I'm not aware of | |
| 17:44:33 | artom | jgwentworth, in this case, the [compute-features-enabled]/config_drive option | |
| 17:44:41 | jgwentworth | right | |
| 17:44:52 | artom | jgwentworth, ah, doesn't ring a bell | |
| 17:44:59 | dansmith | jgwentworth: isn't that per compute node? | |
| 17:45:05 | jgwentworth | yeah, I'm pretty sure there's not | |
| 17:45:15 | jgwentworth | dansmith: oh, yeah I guess it can be | |
| 17:45:28 | artom | jgwentworth, I suppose you could always boot a VM, ssh to it, and check | |
| 17:45:30 | artom | But... | |
| 17:45:45 | artom | You're meant to be using the config option :) | |
| 17:45:49 | jgwentworth | sigh. ok. just thinking, the test could be useful if force_config_drive=False but it's not in the gate so it would be a dupe test | |
| 17:45:56 | artom | And skip tests that need config drive if it's not set | |
| 17:46:07 | jgwentworth | and is only not a dupe in envs where force_config_drive=False | |
| 17:48:09 | jgwentworth | so at least in the gate, it should be disabled by default. maybe it should actually check the opposite, only run if [compute-features-enabled]/config_drive=False | |
| 17:48:39 | jgwentworth | because tempest seems to be using that to mean whether config drive is being forced | |
| 17:49:19 | artom | What does passing config_drive=True on server creation even do? | |
| 17:49:44 | artom | "Indicates whether a config drive enables metadata injection." | |
| 17:49:53 | artom | That's... no the same as "a config drive will be present" | |
| 17:49:55 | artom | Is it? | |
| 17:50:03 | larsks | artom: attaches a config drive to your instance that will be used for getting instance metadata instead of the network metadata service. | |
| 17:50:19 | artom | larsks, see, I had no idea that was user-controllable in that way | |
| 17:50:32 | larsks | Used e.g. by cloud-init for retrieving ssh keys, network confguration, etc. | |
| 17:50:46 | artom | larsks, right, I know what a config drive *is* :) | |
| 17:51:15 | artom | I just didn't think the user had that level of control over its presence | |
| 17:52:31 | artom | API or image metadata properties. Launched instances are not affected by this | |
| 17:52:31 | artom | "When this option is set to true config drive functionality will be forced | |
| 17:52:31 | artom | option." | |
| 17:52:31 | artom | enabled by default, otherwise users can still enable config drives via the REST | |
| 17:52:32 | artom | Oh OK. | |
| 17:53:01 | jgwentworth | ok, I commented accordingly on the review about inverting the logic of the skip | |
| 17:53:03 | mriedem | because not everyone wants to run the metadata api service | |
| 17:53:36 | artom | mriedem, so I heard | |
| 17:53:39 | mriedem | and the metadata api is only as good as the stuff we persist, which isn't everything that you can use, like personality files | |
| 17:54:00 | mriedem | i.e. you might get your goodies on boot but they are gone after resizing | |
| 17:55:28 | dansmith | config drive is a big benefit for edge peeps | |
| 17:55:55 | dansmith | because hitting the metadata api is a trip over the wan, either via http if it's central, or by it to the db which likely is | |
| 17:55:58 | dansmith | configdrive keeps working even if you're isolated | |
| 18:00:10 | openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add nova-audit spec https://review.opendev.org/693226 | |
| 18:02:58 | sean-k-mooney | you can use metadta service if you have isolated neutron networks too | |
| 18:03:14 | sean-k-mooney | but you need to configure neturon to proxy the metadta api via the dhcp agent | |
| 18:03:19 | sean-k-mooney | instead fo the router | |
| 18:03:47 | dansmith | sean-k-mooney: to be clear, I meant partitioned as in...the wan is down | |
| 18:03:48 | sean-k-mooney | but ya config drive woks regardless of the network topology | |
| 18:03:58 | sean-k-mooney | ah right | |
| 18:04:35 | sean-k-mooney | the only draw back with the config drive is that it will not be updated if you attach interface ectra at least while the vm is running | |
| 18:04:50 | sean-k-mooney | in general i works well for what it does | |
| 18:05:21 | artom | jgwentworth, I think gmann and the other tempest folks need to weigh in on https://review.opendev.org/#/c/693096/ - I'm not even sure it's necessary at all | |
| 18:06:50 | jgwentworth | yes, I assumed tempest ppl will review it :) | |
| 18:07:13 | artom | There's... latency, sometimes :) | |
| 18:09:47 | jgwentworth | yeah, I just meant it's a tempest patch so it's gonna be up to them | |
| 18:13:20 | mriedem | dansmith: per usual you updated while i was leaving comments :/ | |
| 18:16:42 | dansmith | mriedem_lunch: sorry man, just fixing pep8 | |
| 18:18:45 | jgwentworth | dansmith: spec looks sane to me and ok for ML input | |
| 18:22:36 | dansmith | jgwentworth: thanks, I'll make some of the changes mriedem_lunch suggested and then blast | |
| 18:38:44 | openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add nova-audit spec https://review.opendev.org/693226 | |