| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-07 | |||
| 20:08:58 | jaypipes | mriedem: I know, right? :) | |
| 20:09:54 | cdent | good night | |
| 20:12:56 | mriedem | jaypipes: efried: want me to do the doc and nova-status stuff from https://review.openstack.org/#/c/385693/ ? | |
| 20:13:25 | efried | mriedem I would welcome that. Were you planning to do it in isolation or add it somewhere in this series? | |
| 20:13:31 | mriedem | isolation | |
| 20:13:35 | mriedem | i'm not touching that mess | |
| 20:13:36 | efried | Perfect | |
| 20:13:39 | efried | Yeah, you shouldn't. | |
| 20:19:13 | melwitt | I don't know why but using self.flags causes the concurrent test fail for the NoopQuotaDriver | |
| 20:19:36 | mriedem | huh, maybe not global enough? | |
| 20:23:41 | melwitt | oh because self.flags doesn't clear the conf override. weird, I thought it would have to so it works with multiple tests | |
| 20:24:29 | efried | melwitt I have a bug for that. | |
| 20:24:30 | efried | Stand by... | |
| 20:24:47 | james_li_ | Hi nova devs, a quick question: is it possible to attach a volume of another tenant to my server? | |
| 20:25:24 | mriedem | james_li_: depends on the policy configuration, but not by default | |
| 20:25:25 | melwitt | gah, what other channel did I accidentally leave by pressing ctrl-w in the wrong window | |
| 20:25:47 | efried | melwitt Sorry, different bug, probably not related: https://bugs.launchpad.net/oslo.config/+bug/1709728 | |
| 20:25:48 | openstack | Launchpad bug 1709728 in oslo.config "CONF.set_override doesn't alias deprecated opts" [Undecided,Confirmed] | |
| 20:26:13 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI https://review.openstack.org/526094 | |
| 20:26:44 | james_li_ | mriedem: cool so its possible if policy enables, e.g. admin user? | |
| 20:27:09 | mriedem | james_li_: the default rule is admin_or_owner, | |
| 20:27:13 | mriedem | if that were changed to @ | |
| 20:27:16 | mriedem | then anything goes | |
| 20:27:56 | mriedem | https://docs.openstack.org/nova/latest/configuration/sample-policy.html | |
| 20:28:03 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI https://review.openstack.org/526094 | |
| 20:28:04 | james_li_ | mriedem: thanks, just wanted to make sure if any code changes are needed for that. | |
| 20:28:09 | mriedem | #"os_compute_api:os-volumes-attachments:create": "rule:admin_or_owner" | |
| 20:28:30 | mriedem | james_li_: shouldn't require code changes | |
| 20:28:36 | mriedem | but policy is a funny thing so you'd have to test it | |
| 20:28:46 | james_li_ | :) | |
| 20:31:51 | melwitt | api samples tests run super ugly under py3 | |
| 20:32:12 | melwitt | lots of warnings emitted to the screen | |
| 20:32:36 | melwitt | and clearing the override didn't seem to help. gdi | |
| 20:33:50 | efried | jaypipes Do we have db constraints or other checks that prevent traits, aggs, and children from existing against a provider UUID that doesn't exist? | |
| 20:34:08 | jaypipes | efried: nope. | |
| 20:34:39 | mriedem | melwitt: just use set_override, it's fine | |
| 20:34:41 | efried | jaypipes So theoretically I could create RP traits first, then create the RP they're associated with? | |
| 20:34:57 | mriedem | and yes on the py35 warnings - i was seeing that in the functional tests earlier today, oslo.contet and oslo.policy warnings | |
| 20:34:57 | efried | Or create an agg with orphan RP UUIDs, then create the RPs associated with those UUIDs? | |
| 20:35:01 | mriedem | we should consider ignoring those | |
| 20:35:35 | jaypipes | efried: yes, manually executing SQL statements. of course, the object layer won't allow you to do that, though. | |
| 20:36:04 | jaypipes | efried: since a TraitNotFound or ResourceProviderNotFound would be raised when attempting to do that via ResourceProvider.set_traits() | |
| 20:36:25 | jaypipes | efried: because there's a lookup of trait ID to names supplied in set_traits() | |
| 20:36:28 | efried | jaypipes oh, okay, phew. So the answer from the perspective of a REST API consumer is that we're strict about that stuff. | |
| 20:36:36 | jaypipes | efried: yes. | |
| 20:36:40 | efried | Good. | |
| 20:36:53 | jaypipes | efried: except for aggregates, which are just UUIDs and we have nothing to "check" against. | |
| 20:37:10 | efried | jaypipes Right, but you can't *associate* a random anonymous UUID with an aggregate? | |
| 20:37:34 | jaypipes | efried: no. but you *can* associated a random UUID to a known resource provider. | |
| 20:37:43 | jaypipes | efried: it's the aggregate UUID we have no way of checking. | |
| 20:37:50 | efried | jaypipes Got it, cool, thanks. | |
| 20:37:55 | jaypipes | pas de probleme | |
| 20:38:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update nova-status and docs for nova-compute requiring placement 1.14 https://review.openstack.org/526505 | |
| 20:38:41 | efried | jaypipes FYI this is coming from a place where I'm refactoring _ensure_resource_provider: In the _create_resource_provider path I can actually skip refresh_aggregate_map | |
| 20:38:44 | mriedem | efried: jaypipes: ^ | |
| 20:38:59 | jaypipes | efried: ack. | |
| 20:39:03 | jaypipes | mriedem: thank you sir. | |
| 20:39:08 | efried | mriedem Thanks | |
| 20:40:34 | mriedem | imacdonn: you can ask but you're going to be hard pressed to find anyone that knows much about libvirt+xen in icehouse in channel right now | |
| 20:40:40 | mriedem | anthonyper is your closest bet | |
| 20:42:03 | imacdonn | yeah, I know ... OK .. so the issue is that nova-compute occassionally gets stuck seemingly in trying to talk to libvirt .. the symptoms are that the resource_tracker no longer reports every minute, and any VM operations that need libvirt fail | |
| 20:42:32 | imacdonn | via guru meditation, I can see that the resource_tracker thread is stuck trying to call libvirt's getLibVersion() | |
| 20:42:51 | mriedem | which is then a call to the hypervisor | |
| 20:42:57 | mriedem | so you're likely deadlocking on something in the hypervisor | |
| 20:42:58 | imacdonn | that call is made through eventlet's thread pooling proxy thingy, which is now holding a lock | |
| 20:43:06 | mriedem | yeah that could also be screwing you | |
| 20:43:16 | mriedem | i'd enable debug logging for libvirt and see if something shows up in there | |
| 20:43:31 | mriedem | or, check to see if things changed around that code since icehouse and see if you need to backport a fix | |
| 20:43:52 | imacdonn | problem is it happens once in a while, and I have like 2k compute nodes ... don't really want to to turn debug on on all of them and wait | |
| 20:44:16 | imacdonn | I've looked around, but not found anything that looks like an obvious related fix | |
| 20:45:03 | imacdonn | I can't tell for sure if it's libvirt hanging on the call, or eventlet getting hung up somehow and not even trying the call | |
| 20:48:43 | mriedem | imacdonn: well, i see this in kilo https://review.openstack.org/#/c/104930/ | |
| 20:49:16 | mriedem | https://review.openstack.org/#/c/104930/12/nova/virt/libvirt/host.py@192 | |
| 20:49:17 | mriedem | so, | |
| 20:49:18 | imacdonn | yeah, that made it "fun" to try to compare bits of of the code to see what might have changed | |
| 20:49:29 | mriedem | keep in mind that anything running in those threads that logs anything could lock you up | |
| 20:49:48 | imacdonn | yes, saw some stuff about that | |
| 20:49:50 | mriedem | so if you have a GMR when things are locked, i'd look for any libvirt driver/host methods in the thread dump, | |
| 20:49:53 | mriedem | and see if those do loging | |
| 20:49:55 | mriedem | *logging | |
| 20:50:22 | imacdonn | GMR is at https://pastebin.com/1jfgdurJ | |
| 20:50:29 | mriedem | because i'm sure we don't do a good job of auditing stuff like that | |
| 20:50:31 | imacdonn | getLibVersion() call at line 659 | |
| 20:51:00 | imacdonn | victim thread waiting to acquire() lock around line 777 | |
| 20:55:36 | imacdonn | seems like getLibVersion() isn't asking much of libvirt ... don't think it'd even have to talk to the hypervisor ... | |
| 20:56:45 | mriedem | well, it's making a connection to libvirt i believe | |
| 20:56:58 | mriedem | so it's not like the libvirt-python package version or something | |
| 20:57:11 | mriedem | nova meeting in 3 minutes | |
| 20:57:42 | imacdonn | it'd make a native library call at least .. dunno if it'd have to connect to libvirtd .. OK, I'll shut up for now ;) | |
| 20:58:03 | mriedem | true yeah | |
| 20:59:21 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (usage) https://review.openstack.org/520603 | |
| 20:59:37 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (inventory) https://review.openstack.org/520613 | |
| 21:00:03 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (resource_class) https://review.openstack.org/520611 | |
| 21:00:03 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (aggregate) https://review.openstack.org/520608 | |
| 21:00:04 | mriedem | melwitt: might want to check out v | |
| 21:00:05 | mriedem | https://bugs.launchpad.net/nova/+bug/1737011 | |
| 21:00:06 | openstack | Launchpad bug 1737011 in OpenStack Compute (nova) "ServerActionsTestJSON.test_reboot_server_hard failed to ssh into instance" [Undecided,New] | |
| 21:00:15 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (trait) https://review.openstack.org/520605 | |
| 21:00:32 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add x-openstack-request-id in API ref https://review.openstack.org/523007 | |
| 21:00:44 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add 'Location' parameters in API ref https://review.openstack.org/521541 | |