Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-02-17
15:00:55 artom How does email fit into that? Is it the heckler?
15:02:00 cdent My brain is starting to break down, and I'm very confused.
15:18:38 des_consolado gah I'm trying to create instances and they just get stuck in the Spawning state and then eventually fail after ages, saying failed to provision instance
15:18:42 openstackgerrit John Garbutt proposed openstack/nova master: POC: improved policy functional tests https://review.openstack.org/435484
15:18:43 openstackgerrit John Garbutt proposed openstack/nova master: POC: add context.check_scope https://review.openstack.org/435485
15:18:50 des_consolado something or other timed out etc
15:18:59 des_consolado I'm tailing every nova log and nothing interesting is there
15:23:52 artom des_consolado, you're probably better off asking in #openstack, as this channel is for development, but it could be waiting for an external event. I've seen these symptoms before where a network-plugged event from Neutron didn't arrive, and the spawn timed out.
15:25:48 des_consolado oh yeah I just read the topic, sorry
15:26:33 des_consolado the other openstack project channels don't seem to make as much of a point about that
15:27:07 des_consolado I also noticed the developers of given projects tend to have all the answers :P
15:29:11 mriedem des_consolado: if it's a virtual interface exception / timeout thing in the n-cpu logs, then yes it's timing out waiting for the external vif plugged callback event from neutron as artom points out
15:29:30 mriedem which probably means something is failing on the neutron side, check the neutron agent health status and logs
15:30:33 sdague johnthetubaguy: I respun with the drop on the delete call
15:32:45 cdent mriedem: was there a change you pushed up as a result of your kvm-on-the-subnode debugging last night? I'd like to look at it if so.
15:33:18 mriedem yeah, sec
15:33:33 mriedem cdent: https://review.openstack.org/#/c/435154/
15:33:38 cdent thanks
15:33:41 mriedem your googling about nested kvm tipped me off
15:33:55 mriedem then i asked in infra and clarkb confirmed the nested kvm issue, and we found the problem in d-g
15:33:57 mriedem mega bash
15:34:00 mriedem what could go wrong
15:34:56 cdent your change was interesting enough to pull jo*go?
15:40:54 cdent mriedem: were you a good boy and search for 'localrc' where $localrc_file ought to be in the rest of the file, or only fix the presenting problem?
15:41:41 mriedem cdent: i pinged him
15:42:34 mriedem cdent: i did
15:43:49 cdent 🍪
16:01:47 mriedem yikes http://logs.openstack.org/11/349011/3/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial-ocata/bee08f9/logs/screen-n-api.txt.gz?level=TRACE#_2017-02-14_22_13_03_345
16:01:55 mriedem Passing insecure dynamic vendordata requests because of missing or incorrect service account configuration.
16:01:57 mriedem all over the logs
16:02:38 melwitt I have noticed that too
16:03:01 melwitt haven't made time to look into it yet
16:04:54 mriedem http://git.openstack.org/cgit/openstack/nova/tree/nova/api/metadata/vendordata_dynamic.py#n52
16:04:58 mriedem we should just log that once
16:05:01 mriedem i'm reporting a bug
16:05:10 mriedem if LP will stop timing out
16:07:22 melwitt nice
16:12:54 sdague mriedem: isn't that feature opt in?
16:13:03 sdague so if it's off, shouldn't we just not log
16:13:14 openstack Launchpad bug 1665693 in OpenStack Compute (nova) ""Passing insecure dynamic vendordata requests because of missing or incorrect service account configuration." warnings all over n-api logs" [Medium,Confirmed]
16:13:14 mriedem https://bugs.launchpad.net/nova/+bug/1665693
16:13:21 mriedem sdague: maybe, i haven't dug in
16:14:38 sdague mriedem: I do agree, it's logging a lot
16:19:40 sdague mriedem: if you get bored - https://review.openstack.org/#/c/435010/ is the code for the project_id enforcement
16:20:07 sdague we get some review on it and I could probably have that in land state for next week. Early blueprint completion for the win.
16:25:33 mriedem sfinucan: did i answer your questions here? https://review.openstack.org/#/c/380433/
16:25:46 mriedem sdague: ok
16:30:22 mriedem sdague: when do you get into ATL?
16:30:26 mriedem mtreinish: ^
16:30:40 sdague mriedem: like 9pm on Sunday
16:31:02 mriedem geez
16:31:35 sdague I can send you my tripit info if you like
16:33:23 sdague mriedem: why, when are you getting in?
16:34:49 sdague I think my reasonable options were this flight, or a super early in the morning one on Sunday, so I went with not having to wake up at unusual times
16:36:22 mriedem sdague: comments in https://review.openstack.org/#/c/435010/
16:36:28 mriedem sdague: 2:45 on sunday
16:41:44 mriedem i know why we're getting the billions of vendordata warnings then, it's because it's hit on each init of the InstanceMetadata object
16:41:51 mriedem which is every server that gets created with config drive
16:44:23 mriedem sdague: so can we just pick one of these options to tell if we can safely assume dynamic vendordata is just not configured to work? http://logs.openstack.org/11/349011/3/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial-ocata/bee08f9/logs/screen-n-api.txt.gz#_2017-02-14_21_31_00_838
16:44:33 mriedem like auth_section or auth_type?
16:44:35 mriedem jamielennox: ^
16:45:39 mriedem sounds like auth_section is optional
16:45:41 mriedem " auth_section: the group from which further auth plugin options should be taken. If section is not provided then the auth plugin options will be taken from the same group as provided in the parameters. "
16:45:58 mriedem but auth_type has to be set right?
16:46:04 mriedem lbragstad: ^?
16:48:05 mriedem well, i guess it doesn't matter, we can just log it once
16:48:18 mriedem we still make the requests, they just aren't with the service token
16:55:27 lbragstad mriedem yes - i believe auth_type needs to be set
17:13:05 openstackgerrit Chris Dent proposed openstack/nova master: Update placement_dev with info about new decorator https://review.openstack.org/435539
17:13:58 sdague mriedem: of course, by tightening up the exception handling, I can't use the testing exception :)
17:16:33 cdent Is there a rule of thumb on when lack of response on a bug report is cause for marking invalid?
17:17:50 sdague cdent: if you set it to Incomplete, and remove assignee, it will expire in 60 days (IIRC)
17:18:11 cdent thanks
17:26:30 ildikov johnthetubaguy: etherpad for the Cinder-Nova session next week: https://etherpad.openstack.org/p/ptg-pike-cinder-nova-session
17:27:03 ildikov johnthetubaguy: if you can get there please add a few bullet points for discussion, I will try to look into it too
17:27:19 ildikov johnthetubaguy: also safe travels and see you next week :)
17:27:30 johnthetubaguy ildikov: ah, cool, will try get into that
17:27:49 ildikov johnthetubaguy: thanks much
17:28:03 ildikov johnthetubaguy: would be great to have a somewhat guided discussion
17:29:27 johnthetubaguy ildikov: +1
17:31:25 openstackgerrit melanie witt proposed openstack/nova master: Skip soft-deleted records in 330_enforce_mitaka_online_migrations https://review.openstack.org/435546
17:42:31 zzfoo Unable to acquire lock on `/opt/stack/data/nova/instances/locks/nova-fd2836e4eef36d5483a37a5000a42a72fb95ea7c` due to [Errno 37] No locks available
17:42:31 zzfoo Hi, I'm trying to get live migration working. I have shared store setup and working with NFS though when i do a live migrate I get the following error in the nova-compute logs: error:
17:43:04 zzfoo The weird, thing is this lock file does exist
17:43:43 zzfoo any ideas why this is occurring? I also set the libvirtd group id to be the same across all the nodes and ls in this shared mount shows the same group ownerships
18:01:48 diana_clarke mriedem1: Do you know approximately how many people will be at the nova PTG?
18:30:54 mriedem diana_clarke: i do not
18:32:57 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Only log vendoradata_dynamic auth warning once https://review.openstack.org/435563
18:34:44 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add additional-default-policy-roles spec https://review.openstack.org/427872
18:34:45 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add policy-remove-scope-checks spec https://review.openstack.org/433037
18:36:26 mriedem a shiny nickel to whoever can figure out why my unit test is failing in ^
18:37:36 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add policy-docs spec https://review.openstack.org/433010
18:38:10 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add policy-remove-scope-checks spec https://review.openstack.org/433037
18:38:16 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add additional-default-policy-roles spec https://review.openstack.org/427872
18:39:47 sdague mriedem: because something else managed to set that global already
18:39:57 mriedem gah
18:40:10 mriedem ok i knew i needed to mock that out at some point in here too
18:40:23 mriedem sdague: i owe you a nickel
18:40:25 mriedem :)
18:50:58 openstackgerrit Pushkar Umaranikar proposed openstack/nova master: Placement api: set custom json_error_formatter in resource_class https://review.openstack.org/430463

Earlier   Later