Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-07
20:26:44 james_li_ mriedem: cool so its possible if policy enables, e.g. admin user?
20:27:09 mriedem james_li_: the default rule is admin_or_owner,
20:27:13 mriedem if that were changed to @
20:27:16 mriedem then anything goes
20:27:56 mriedem https://docs.openstack.org/nova/latest/configuration/sample-policy.html
20:28:03 openstackgerrit Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI https://review.openstack.org/526094
20:28:04 james_li_ mriedem: thanks, just wanted to make sure if any code changes are needed for that.
20:28:09 mriedem #"os_compute_api:os-volumes-attachments:create": "rule:admin_or_owner"
20:28:30 mriedem james_li_: shouldn't require code changes
20:28:36 mriedem but policy is a funny thing so you'd have to test it
20:28:46 james_li_ :)
20:31:51 melwitt api samples tests run super ugly under py3
20:32:12 melwitt lots of warnings emitted to the screen
20:32:36 melwitt and clearing the override didn't seem to help. gdi
20:33:50 efried jaypipes Do we have db constraints or other checks that prevent traits, aggs, and children from existing against a provider UUID that doesn't exist?
20:34:08 jaypipes efried: nope.
20:34:39 mriedem melwitt: just use set_override, it's fine
20:34:41 efried jaypipes So theoretically I could create RP traits first, then create the RP they're associated with?
20:34:57 mriedem and yes on the py35 warnings - i was seeing that in the functional tests earlier today, oslo.contet and oslo.policy warnings
20:34:57 efried Or create an agg with orphan RP UUIDs, then create the RPs associated with those UUIDs?
20:35:01 mriedem we should consider ignoring those
20:35:35 jaypipes efried: yes, manually executing SQL statements. of course, the object layer won't allow you to do that, though.
20:36:04 jaypipes efried: since a TraitNotFound or ResourceProviderNotFound would be raised when attempting to do that via ResourceProvider.set_traits()
20:36:25 jaypipes efried: because there's a lookup of trait ID to names supplied in set_traits()
20:36:28 efried jaypipes oh, okay, phew. So the answer from the perspective of a REST API consumer is that we're strict about that stuff.
20:36:36 jaypipes efried: yes.
20:36:40 efried Good.
20:36:53 jaypipes efried: except for aggregates, which are just UUIDs and we have nothing to "check" against.
20:37:10 efried jaypipes Right, but you can't *associate* a random anonymous UUID with an aggregate?
20:37:34 jaypipes efried: no. but you *can* associated a random UUID to a known resource provider.
20:37:43 jaypipes efried: it's the aggregate UUID we have no way of checking.
20:37:50 efried jaypipes Got it, cool, thanks.
20:37:55 jaypipes pas de probleme
20:38:40 openstackgerrit Matt Riedemann proposed openstack/nova master: Update nova-status and docs for nova-compute requiring placement 1.14 https://review.openstack.org/526505
20:38:41 efried jaypipes FYI this is coming from a place where I'm refactoring _ensure_resource_provider: In the _create_resource_provider path I can actually skip refresh_aggregate_map
20:38:44 mriedem efried: jaypipes: ^
20:38:59 jaypipes efried: ack.
20:39:03 jaypipes mriedem: thank you sir.
20:39:08 efried mriedem Thanks
20:40:34 mriedem imacdonn: you can ask but you're going to be hard pressed to find anyone that knows much about libvirt+xen in icehouse in channel right now
20:40:40 mriedem anthonyper is your closest bet
20:42:03 imacdonn yeah, I know ... OK .. so the issue is that nova-compute occassionally gets stuck seemingly in trying to talk to libvirt .. the symptoms are that the resource_tracker no longer reports every minute, and any VM operations that need libvirt fail
20:42:32 imacdonn via guru meditation, I can see that the resource_tracker thread is stuck trying to call libvirt's getLibVersion()
20:42:51 mriedem which is then a call to the hypervisor
20:42:57 mriedem so you're likely deadlocking on something in the hypervisor
20:42:58 imacdonn that call is made through eventlet's thread pooling proxy thingy, which is now holding a lock
20:43:06 mriedem yeah that could also be screwing you
20:43:16 mriedem i'd enable debug logging for libvirt and see if something shows up in there
20:43:31 mriedem or, check to see if things changed around that code since icehouse and see if you need to backport a fix
20:43:52 imacdonn problem is it happens once in a while, and I have like 2k compute nodes ... don't really want to to turn debug on on all of them and wait
20:44:16 imacdonn I've looked around, but not found anything that looks like an obvious related fix
20:45:03 imacdonn I can't tell for sure if it's libvirt hanging on the call, or eventlet getting hung up somehow and not even trying the call
20:48:43 mriedem imacdonn: well, i see this in kilo https://review.openstack.org/#/c/104930/
20:49:16 mriedem https://review.openstack.org/#/c/104930/12/nova/virt/libvirt/host.py@192
20:49:17 mriedem so,
20:49:18 imacdonn yeah, that made it "fun" to try to compare bits of of the code to see what might have changed
20:49:29 mriedem keep in mind that anything running in those threads that logs anything could lock you up
20:49:48 imacdonn yes, saw some stuff about that
20:49:50 mriedem so if you have a GMR when things are locked, i'd look for any libvirt driver/host methods in the thread dump,
20:49:53 mriedem and see if those do loging
20:49:55 mriedem *logging
20:50:22 imacdonn GMR is at https://pastebin.com/1jfgdurJ
20:50:29 mriedem because i'm sure we don't do a good job of auditing stuff like that
20:50:31 imacdonn getLibVersion() call at line 659
20:51:00 imacdonn victim thread waiting to acquire() lock around line 777
20:55:36 imacdonn seems like getLibVersion() isn't asking much of libvirt ... don't think it'd even have to talk to the hypervisor ...
20:56:45 mriedem well, it's making a connection to libvirt i believe
20:56:58 mriedem so it's not like the libvirt-python package version or something
20:57:11 mriedem nova meeting in 3 minutes
20:57:42 imacdonn it'd make a native library call at least .. dunno if it'd have to connect to libvirtd .. OK, I'll shut up for now ;)
20:58:03 mriedem true yeah
20:59:21 openstackgerrit Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (usage) https://review.openstack.org/520603
20:59:37 openstackgerrit Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (inventory) https://review.openstack.org/520613
21:00:03 openstackgerrit Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (resource_class) https://review.openstack.org/520611
21:00:03 openstackgerrit Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (aggregate) https://review.openstack.org/520608
21:00:04 mriedem melwitt: might want to check out v
21:00:05 mriedem https://bugs.launchpad.net/nova/+bug/1737011
21:00:06 openstack Launchpad bug 1737011 in OpenStack Compute (nova) "ServerActionsTestJSON.test_reboot_server_hard failed to ssh into instance" [Undecided,New]
21:00:15 openstackgerrit Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (trait) https://review.openstack.org/520605
21:00:32 openstackgerrit Takashi NATSUME proposed openstack/nova master: [placement] Add x-openstack-request-id in API ref https://review.openstack.org/523007
21:00:44 openstackgerrit Takashi NATSUME proposed openstack/nova master: [placement] Add 'Location' parameters in API ref https://review.openstack.org/521541
21:01:14 openstackgerrit Takashi NATSUME proposed openstack/nova master: [placement] Fix getting placement request ID https://review.openstack.org/523606
21:01:57 openstackgerrit Takashi NATSUME proposed openstack/nova master: [placement] Add functional tests for resource class API https://review.openstack.org/524506
21:02:09 openstackgerrit Takashi NATSUME proposed openstack/nova master: [placement] Add functional tests for traits API https://review.openstack.org/524094
21:03:03 openstackgerrit Takashi NATSUME proposed openstack/nova master: [cellv2] Improve getting BDMs in multiple cells https://review.openstack.org/521400
21:03:28 openstackgerrit melanie witt proposed openstack/nova master: Add API and nova-manage tests that use the NoopQuotaDriver https://review.openstack.org/526270
21:03:28 openstackgerrit melanie witt proposed openstack/nova master: Follow up on removing old-style quotas code https://review.openstack.org/524234
21:13:44 openstackgerrit Takashi NATSUME proposed openstack/nova master: List/show all server migration types (1/2) https://review.openstack.org/430608
21:14:42 openstackgerrit Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483
21:16:31 melwitt mriedem: is that different than our usual sporadic ssh timeouts?
21:19:33 mriedem melwitt: i don't think ssh timeouts are all that sporadic anymore
21:19:35 mriedem andreaf: ^?
21:20:19 melwitt okay, I saw one recently that I had to recheck so I thought they were still going on. I guess I should look at that one and compare
21:21:38 mriedem i don't see cloud-init run at all
21:24:23 mriedem force_config_drive = True
21:24:32 mriedem so we force a config drive to inject keys and such
21:27:01 mriedem we don't even get to the point of rebooting the serer
21:27:02 mriedem *server
21:27:08 mriedem it's trying to ssh into the guest to check uptime before that
21:27:10 mriedem and that's what fails

Earlier   Later